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Robert Price 


Robert Price (S’48—A’54) was born on July 7, 1929, in 
West Chester, Pennsylvania. He received the A.B. degree 
in physics from Princeton University, Princeton, N. J., 
in 1950, and the Se.D. degree in electrical engineering 
from the Massachusetts Institute of Technology, Cam- 
bridge, Mass., in 1953. 

His doctoral study was started at the Research Labo- 
ratory of Electronics of M.I.T. and continued later at the 
Lincoln Laboratory, Lexington, Mass, While at the Re- 
search Laboratory of Electronics, he held an Industrial 
Fellowship. His dissertation was concerned with the appli- 
cation of modern statistical methods to the problems of 
communicating over a channel characterized by time- 
varying multipath and noise. Upon graduation from 
M.I.T., he spent a year as a Fulbright Fellow, engaged 


in radio-astronomy at the Radiophysics Division of the 
Commonwealth Scientific and Industrial Research Organ- 
ization in Sydney, Australia. Returning to the M.I.T. 
Lincoln Laboratory in 1954, he again became interested 
in multipath channels, and is a co-originator of the Rake 
antimultipath communication system. Recently he has 
been concerned with the Venus radar experiment. 

His interests lie in both the theoretical and experimental 
aspects of statistical communication theory, and he has 
written on detection theory, signal statistics, and iono- 
spheric measurement techniques, in addition to papers 
dealing with operational systems. 

Dr. Price isa member of Phi Beta Kappa, Sigma Xi, and 
the Franklin Institute. He also recently has been elected 
a member of the U.S.A. Subcommission 6.1 of URSI. 


that modern statistical methods have not yet made 
the bold impact on communication technology that 
fas predicted with enthusiasm a few years ago. If we are 
ndid, we must admit that today it is a rare communi- 
tion system, either in the field or presently in fabrication, 
hat exhibits a “new look” directly attributable to our 
orts. One need only scan the TRANSACTIONS of our 
ter Professional Group on COMMUNICATIONS SYSTEMS to 
be that contemporary system design is drawn in the main 
-om the pre-Wiener era. 

To be sure, there are a number of techniques in current 
e that, while dating back some years, provide good 
proximations to statistical optimality. The improve- 
hent in noisy signals achieved through ‘common sense” 
fesign of filters in the frequency domain, the use of pulse 
ode modulation to obtain reliable digital communication, 
nd the reduction of digital system error rates realized 
hrough the use of matched filters, are examples of current 
rractice compatible with the precepts of our theories. It 
onfidently can be claimed further that modern statistical 
malysis has been able, in many cases, to show the upper 
‘mits of attainment, and thereby has forestalled untold 
husings and profitless experiment. In this respect the 
ecognition of the governing roles of irreducible error, 
hannel capacity, and energy-to-noise-density ratio has 
a of inestimable value. 


| YHERE is a tendency in our field to be apologetic 


These accomplishments are of an essentially negative 
aracter, however, and as such we cannot take undue 
ride in hon What we really should look forward to, or 
o we are told, are the coming positive applications of 
tatistical theory. When certain difficult problems are 
inally resolved, there will appear systems, or at least 
hajor parts of systems, whose performance will be radi- 
ally improved over that we know today. Certainly there 
ill be at least one advance of this magnitude when 
resent studies on practical coding and decoding schemes 
ave progressed to the point where the Fundamental 
"heorem of Information Theory can be realized in an 
jperational system. 

While we are waiting to revolutionize the communi- 
ation field, however, I would like to argue that the 
asights and glimpses of fundamental truths already 
ained are of great intrinsic merit. A philosopher might 
ontend that these truths are our only real achievements, 


* Lincoln Laboratory, Massachusetts Institute of Technology, 


exington, Mass. 
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and that what improvement we may introduce in this 
or that communication system is of relatively lesser sig- 
nificance in the eternal scheme of things. 

In illustration of such fundamental truths, I would 
first select the truly remarkable Fundamental Theorem. 
Shannon’s discovery that errors can be made negligible 
while preserving a positive information rate over a noisy 
channel is certainly a major triumph. A second basic 
concept, which underlies the I'undamental Theorem but 
appears to be of significance in itself, is that of informa- 
tional entropy. Although informational entropy does not 
strictly obey the conservational and invariance laws 
usually expected of a basic entity, no rival measure has 
come forth to dispute its preeminence. Both here and 
in Russia, the entropy measure has been found closely 
linked with filtering theory, and entropy could conceiv- 
ably play an important role in nonlinear filtering studies. 
It would seem, in any case, that the last word has not 
yet been said about the properties of informational en- 
tropy. A third deep insight resides in Woodward’s obser- 
vation that all the information in an effect about its 
possible causes is contained in the set of a posteriori 
probabilities relating the effect back to the causes. This 
simple notion is a powerful one, for it gives understanding 
of why the likelihood ratio is at the root of detection 
and decision theory. 

We may hope that yet further prime truths await 
discovery, and that the future will see better unity be- 
tween our sub-disciplines of filtering theory, information 
theory, detection theory, and the analysis of signal 
statistics. It seems to me that there is just as much need 
for fresh ideas in our field as there is for the successful 
practical application of our present store of knowledge. 

What I am pleading here is nothing more than the 
importance of basic research, and the dignity of knowl- 
edge for its own sake. Although statistical communication 
theory was largely inspired in its origins by the hope of 
improving communication and related systems, there is 
no reason why it must be tied entirely to its early pur- 
pose. Needless to say, I am by no means advocating the 
abandonment of the communication industry; on the 
contrary, cross-fertilization between pure and applied 
research in our field is bound to reap dividends for both. 

Lest someone expect a sequel to appear under the title 
of “Information Theory, Photosynthesis, and Religion,” 
I want to emphasize that truly basic studies are thorough 
and well-grounded. Their rigor is not sapped by casual 
overextension into a conglomeration of neighboring fields. 
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At the other extreme, basic research is not served by 
tedious examination of one very restricted special case 
after another, unless it be with the purpose of seeking 
a common unity. 

In conclusion, then, past accomplishments can be re- 


garded with satisfaction but not with complacence. Ther 
will always be the continuing obligation to improve 
practical systems. But let us never fail to keep delving 
for those deeper, unifying secrets that surely yield the 
ultimate rewards. 


| Summary—The main body of this paper reports on experi- 
ental results in sequential detection. In particular, it is shown 
at the Wald Theory of Sequential Analysis agrees well with 
t periment for the important case of Bernoulli Detection even when 
4 excess over the boundaries at the termination of an experiment 
‘neglected. The design of the experiments, as well as the experi- 
ental apparatus, are also discussed. Experimental curves of the 
berating Characteristic (OC) and Average Sample Number 
{SN) Functions for several sets of parameters are given. 

A publication relative to the main body of this paper! is sum- 
arized. The results of this publication are used in the Addendum, 
study the resonant properties of the exponential class of se- 
tential detectors. The practical use of these detectors for param- 
er estimation is discussed. 


I. InrRoDUCTION 


NHE experimental results described here serve 
primarily two purposes. First, it is well known 
that for the discrete independent case the charac- 
sristics which describe the performance of a sequential 
etector are only approximate. The errors are introduced 
ecause the excess over the decision boundaries at the 
brmination of an experiment is neglected. This introduces 
‘rors in the OC Function as well as in the ASN Function. 
fal has obtained upper and lower bounds on these 
inctions which are expected to be quite satisfactory for 
wreshold signals, for example, when a large sample 
umber is required, on the average, for the termination of 
1 experiment. It is our aim to study these characteristics 
xperimentally. 

The conventional way of constructing a sequential 
xperiment is to measure the ASN and OC functions for 
arious sets of threshold parameters (4, 6:1, a, 8) as a 
inction of the parameter @ corresponding to a one 
arameter family of distributions., The four-dimensional 
arameter point (4, 6, a, 8) defines a sequential test” 
strength (a, 6). The one parameter family of random 
ariables is then generated by varying the parameter 0. 
ach random variable in the family generates one point 
a the OC Function, L(@), and on the corresponding 
SN Function. However, it has been shown by the 
thor’ that for a very general class of one parameter 
milies of distributions of practical and physical interest, 
sequential test is defined by a set of three transformations 
On 0, a Gh, 0 (Go, 0), a, 8), € (Os, 9:)}. Thus, there 


* Manuscript received by the PGIT, June 3, 1958. This re- 
arch was supported by the United States Air Force through the 
ffice of Scientific Research of the Air Research and Development 
‘ommand. 

+ Electronic Communications, Inc., Timonium, Md. 

1H. Blasbalg, ‘Transformation of the fundamental relation- 
nips in sequential analysis,” Annals of Math Statistics, vol. 28, 
p. 1024-1028; December, 1957. 

2A. Wald, “Sequential Analysis,’ John Wiley and Sons, Inc., 
flew York, N. Y:; 1947. 
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is a degeneracy in sequential analysis with respect to this 
class of one parameter families of distributions. Hence, 
there exists an infinity of parameter points (6, 61, a, 8) 
all of which have the same sequential detector defined by 
a three-dimensional parameter point (a’, b’, c’). It is 
therefore appropriate to measure the ASN and OC 
Functions as a function of @ for a given set (a’, 0’, c’). 
Since the latter point is of general interest and of particular 
importance in this paper, we will summarize very briefly 
the results found earlier.’ 
For the class of distribution functions given by 


dP(x, 6) = exp [r(#) A(x) + s(@)B(x)] dw(z), (1) 


it is shown that a set of three transformations 


log = + log * = 
OO im 
log : + log B 
Tee een a: = (3) 
roe r(8o) : 
c! s(6;) = S( 9) (4) 


* TO) team) oe 


completely define the sequential probability ratio test 
for testing the hypothesis H, that @ < 4) against H, that 
6 > 6,(0, > 4). When the observer specifies the threshold 
parameters 6) and 6,, corresponding to the hypothesis 
H, and H, and the strength (a, 8) of the test, he specifies 
the three transformations and hence the sequential test. 
However, there is an infinity of parameter points (4, 4:, 
a, 8) which satisfy the same transformations and hence 
the same sequential test. 

For a given a priori probability é, it is known that the 
sequential probability ratio test is optimum® in the sense 
that the ASN Function at the parameter point (4, 41, a, 8) 
is less or equal to the ASN Function for any other sequen- 
tial test. Since these parameter points specify the (a’, 
b’, c’) transformations of the sequential probability 
ratio test, this test is also optimum for a given set 
(a’, b’, c’). Since for a given a prior? probability & and 
a given set (a’, b’, c’) there exists an infinity of param- 
eter points (4, 6,, a, 8) which satisfy the transformations, 
we conclude that a given sequential test is optimum at an 
infinity of parameter points (4, 6:, a, 8) which satisfy 
the given transformations. 


3D. Blackwell, and M. A. Girshick, “Theory of Games and 
Statistical Decisions,’ John Wiley and Sons, Inc., New York, 
N. Y.; 1954. 

4M. A. Girshick, “An extension of the optimum property of 
the sequential probability ratio test,’ Annals of Math Statistics, 
vol. 29, pp. 288-290; March, 1958. 
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In a coordinated paper,’ it is shown that the one 
parameter family of distributions in (1) is, from a practical 
point of view, the only family which exhibits this de- 
generacy with respect to sequential analysis. This family 
of distributions includes the very important and well- 
known exponential class of distributions which contaims 
the Gaussian, Rayleigh, Bernoulli, Poisson, ete. 

The optimum decision regions for this general class in 
terms of the (a’, b’, c’) transformations are given by the 
simple relationships, 


AG) 4c’ BG) Sa A (5) 
and, 
», AG) Ac’ DS Bay <b = AG. (6) 


The arrows indicate the hypotheses to which the decision 
regions belong. 
The OC Function is given by the parametric equations 


a’ =a 1 

Ly = a re (7) 
U Se 

Hele ee eal = ths (8) 


and the ASN Function is given by 


b/L(u) + a’[1 — L(u)] 


P= AG RCE ) 
At the value 6 = 6 for which, 
E,{ A(z)} + c’H,{B(a)} = 0, (10) 
which corresponds to wu = 1, we have, 
on! 
LO) Ss =a 11 
and, 
Byln) == (12) 


By {{A(a) + BP} 


The parameter uw ranges over the positive half of the 
real line (0 < wu < @), and H, is the conditional expected 
value operator of the random variable z. 

Since the one parameter family of Bernoulli Distribu- 
tions belongs to the general class for which these results 
apply and since this class is of practical importance in 
detection problems,” we will measure the OC and ASN 
Functions for Bernoulli Detection. In fact it was shown 
by the author’ that the Bernoulli case exhibits this 
degeneracy. The general expressions for the bounds on 
these functions obtained by Wald’ can be easily special- 


5L. J. Savage, “When different pairs of hypotheses have the 
same family of likelihood-ratio test regions,’ Annals of Math Sta- 
tistics, vol. 28, pp. 1028-1032; December, 1957. 

6H. Blasbalg, ‘“The relationship of sequential filter theory to 
information theory and its application to the detection of signals 
in noise by Bernoulli trials,’ IRE Trans. on INFORMATION THEORY, 
vol. IT-3, pp. 122-131; June, 1957. 

7A. Wald, op. cit., pp. 161-179. 
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ized for the Bernoulli case. In order to obtain the char- 
acteristics, we are forced into the practical design of a 
Bernoulli Sequential Detector for which the theory is 
developed. For the sake of simplicity, we will mvestigate 
the OC and ASN Functions for the case a’ = —b’. This 
corresponds to the case where the type I and type IL 
errors a and 8 are equal. 


Il. Design oF AN EXPERIMENTAL BERNOULLI 
SEQUENTIAL DerEectoRr 


For the Bernoulli case we have slightly modified the 
choice of the transformations in order to simplify the 
desired form for the decision regions. These transformations 
are related in a very simple way to the general transforma- 
tions given in (2), (3), and (4). It is easy to show that the 
relationship between (a’, b’, c’) and the (a, b, ¢) trans- 
formations used here is 


eee ee LP 
ee sh” 
(ea 
: e+1’ 
Pijests ot tea 
Perse 


In an earlier paper’ it was shown that the decision 
regions of the Bernoulli Sequential Detector are given by 


oe eiegy ee oi > Ho (13) 
a n 
> f= 
where 
k,, = number of ones measured in n Bernoulli trials 
n = total number of ones and zeros measured. 
The constants (a, b, c) are given as, 
log : = B 
a= ; (15) 
log Po 
1 a Pi 
log i B S 
b= eae (16) 
log 
2 oe 
log fa 
Oa acer S ’ (17) 
log _ 
il a Pi 
where, 


Pp: = probability of measuring a ‘one’ when the 
hypothesis H, is true. 


(as ) 


Po = probability of measuring a “one” when the 
hypothesis H, is true. 
a = probability of accepting H, when H, is true. 


8 = probability of accepting H, when H, is true. 
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In (18) and (14), if we let n = k, + L, where l, is the 
umber of zeros in 7 trials, then the decision regions take 
he form, 


v a S 0 
k€ een (18) 


kce—-l,—-a>0-d, 


n our case, we will make (a, b, c) integers. Since k,, and 
_are also integers, the physical realization of the Bernoulli 
Yetector is a preset reversible binary counter. The 
eversible binary counter is preset to the number |b|. 
ach time a one is measured c¢ ones are read into the 
ounter in the forward direction. Each time a zero is 
reasured a single count is fed into the binary counter in 
he reverse direction. If the counter returns to zero 
efore reaching the threshold a + \b|, the hypothesis 
J, is accepted that the parameter of the Bernoulli Distri- 
ution is p < po. On the other hand, if the counter reaches 
e number a + |b| prior to reaching zero, the hypothesis 
J, is accepted that the Bernoulli parameter is p > pu. 
rom (18) it is seen that whenever H, is accepted there is 
o excess over the boundaries at the termination of an 
periment with the acceptance of H,. Thus for the range 


f parameters p < po which almost always leads to the 
eceptance of Ho, we expect very close agreement between 
eory and experiment. 
It is clear from this discussion that the binary counter 
aust have a capacity, 
H = log, (a + | b |) bits 
bee ee (19) 
= logs = E 
log esis 
il = Pi a 
Nhen a = 8, a = |b|, and we have 
H=1+4+  log,a bits 
ee a 
= 1 + log, 
log 2: ae Bo 
I = Pi 
log . 
gl et NOC eg eae ee) a <l.. (20) 
log = 1G) 
IL = Pi_} 


t (a, \b|). The behavior of the capacity as a function of 
e threshold parameters (po, 71, a, 8) is also indicated. 


In that same paper’ it is shown that the OC Function 
br the Bernoulli Detector is given by the parametric 
uations, eis 


(21) 


be 0 Frae <0; 
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and 
(0) wey OSU S @. (22) 
When u = 1, we have 
Ibi iar SAS, (23) 
1 
OVE eres (24) 
Tor the special case considered here, a = |b|, (a = 8), 
we have 
us 
L,(u) — ye TE I (25) 
and 
END) ss (26) 
The ASN Function is given by 
: _ bLw) + all — Lw) 
Ey uy(n) a ( a c)p(u) = 1 (27) 
When wu = 1, we have 
a| b , 
Byoy(n) = Ob (28) 
Yor the special case a = |b| (a = B), we have 
i ee eal 
J ORTON) ri (1 = c)p(u) iad 1 (29) 
2 
< a 
Lan) =~ (30) 


C 


The OC Function gives the probability of accepting 
the hypothesis H, when p is the parameter of the Bernoulli 
Distribution, while the ASN Function gives the average 
number of samples required for the termination of the 
Sequential Test when p is the parameter of the Bernoulli 
Distribution. Thus, from the experimental point of view, 
it is required to generate a family of Bernoulli random 
variables designated by the parameters, p, and to measure 
the OC and ASN Functions L,(w) and L,,,., (n), respectively. 

Eqs. (21), (22) and (29) show theoretically that all of 
these functions are related to each other through the 
parameter wu. The parameter is, however, not physically 
measurable directly and is of no interest except for ana- 
lytical purposes. The quantities p, L,, H’,(n) are observable 
directly. 

Let M be the total number of sequential experiments 
required to determine one point on the ASN Function 
(for example, for a given parameter p). Thus for a given 
set of constants (a, b, c) of the detector and for a given 
input random variable of parameter p, M decisions are 
made or equivalently a total of M crossings of the thres- 
holds a and b are counted. Let N(J/) be the total number 
of samples (sum of zeros and ones) measured in the M 
sequential experiments. Thus the empirical ASN Funetion 
is given by 
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E,(n, M) = N(M)/M. (31) 


The empirical ASN Function approaches the theoretical 
ASN Function with probability one, as M is allowed to 
increase indefinitely since the experiments are independent. 
Thus, for large IW, the empirical ASN Function is for all 
practical purposes equal to the theoretical ASN Function. 

We generate the Bernoulli Family of Distributions by 
sampling a band-limited low-pass white Gaussian noise 
source at sample points separated in time by an amount 
much greater than the reciprocal of twice the bandwidth. 
This insures us that the samples are, for all practical 
purposes, statistically independent. If o is the parameter 
of the noise family, then a Bernoulli random variable can 
be generated by measuring the number of ones and zeros 
at the sample points at the output of a slicer which has 
the following nonlinear characteristic; 


G[x(o),e]=1 a(o) > 2° 
0 ie CaO hes 


(32) 


I 


Thus for a given slicing threshold x°, we can generate a 
family of Bernoulli Sequences by varying the parameter o 
of the noise source. (Note that o represents a labeling of 
the noise ensembles. It can be a dial control in a noise 
source or a gain control in an amplifier driven by a noise 
source or an ensemble of band-limited noise sources of 
different powers, etc.) It is not necessary to measure o 
but only the number of zeros and ones at the output of 
the operator G[z°]. Actually the only necessary a priort 
information is that the samples are statistically inde- 
pendent. 

The frequency of ones in a total of M sequential experi- 
ments is given by 


v(M) = Y(N)/N(M) (33) 
where 
Y(N) = number of ones in NV (J) samples. 
For N(M) large, it is known* that, 
Prob. [»,(M) > p] 1 as Mo om. (34) 


That is, the empirical probability is certain to converge 
to the mathematical probability as the number of samples 
is increased indefinitely. We can increase the total number 
of samples by increasing the number of Sequential Experi- 
ments and hence improve the approximation to the 
theoretical case. By increasing the number M we also 
improve the approximation to the ASN Function. 

When the parameter p, — po K 1, the ASN Function 
is sharply peaked. However, when this is the case, the 
number of samples required in the parameter range 
Po < p < p, for termination is large. Hence, we auto- 
matically gain precision in the measurement of the Ber- 
noulli parameter p when it is most required. At the tails 
of the ASN Function which correspond to p < po, p > p:, 
the total number of samples N(M) is reduced, therefore 
giving less precision in the measurement of p. However, 


8 M. Loeve, ‘“‘Probability Theory,’ D. Van Nostrand Co., Inc., 
New York, N. Y.; 1955. 
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in this region the behavior of the ASN Function is no 
critical (and not very important from a practical view 
point). We can therefore afford to lose some precision in p 

Another important point is that p and W,(n) are meas 
ured simultaneously. This not only results in a time saving 
but puts less restriction on the stability of the entir 
measuring apparatus. We require stability only over the 
measurement time interval corresponding to one point 
This appears to us of great practical umportance. The 
points on the ASN curve are not taken at equal intervals 
but actually at random intervals. 

The OC Function is also measured simultaneously witk 
the ASN Function. This function gives the probability 
of accepting the hypothesis Hy, when p is the Bernoull 
parameter. Thus to measure this function, it is necessary 
to count the number of times in M sequential experiments 
that the reversible binary counter (discussed in Section ID) 
returns to zero. Let L,(M) be this number. Then 

LM) = (35) 
where a(M) is the number of times in M experiments 
that the hypothesis H, is accepted. Of course, M — a(M) 
is the number of times that H, is accepted. Since the 
experiments are independent, we know that the empirical 
OC Function of (35) is certain to approach the mathe- 
matical OC Function at each point as the number of 
sequential experiments is allowed to increase indefinitely. 

For each parameter o of the noise source, we measure 
the Bernoulli parameter p and the corresponding point 
on the OC and ASN Functions simultaneously. We 
emphasize again that this approach increases the speed 
of measurement and reduces the requirements on the 
stability of the operations. The variable precision in 7 
automatically adjusts itself so that more precision is 
obtained when required and less precision when not 
required. or example at the indeterminate point p’ = 
1/(c + 1), which is an important point in the measure- 
ment, the ASN Function is a maximum, hence the total 
number of samples N(M/) measured for the estimate of 
p’ is also a maximum. We emphasize this point as it is 
well-known that the functional behavior of a sequential 
test is the same for a very general class of one parametel! 
family of distributions that are of practical interest. Thus 
the type of measurements described here can also be made 
on other statistical variables with the same advantages 
as for Bernoulli statistics. 


IV. FuncrionaL Buock D1AGRAM oF A SEQUENTIAL 
BERNOULLI DETECTOR AND DISCUSSION OF EXPERI- © 
MENTAL SYSTEM OPERATION 


Fig. 1 is a block diagram of the Bernoulli Sequential 
Detector and the experimental apparatus necessary tc 
obtain the OC and ASN Functions. A detailed description 
of the logical operation of the system will now be presented. 

The primary source for generating the Bernoulli 
Sequence is a noise source whose output is a Gaussiar 
signal having a flat power spectrum. The output at 
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| 1—Sequential Bernoulli filter and experimental apparatus. 


hinal 1 is fed into a slicer whose dynamic charac- 
stic is given by (32). That is, the slicer output is a 
om square wave of unit height corresponding to the 
e intervals during which f(t) is above the threshold. 
nce, at terminal 2 the positive pulses correspond to 
ones in the sequence (x; => 2%) and at terminal 3, 
positive pulses correspond to the zeros in the sequence 
<a). The sequence of zeros and ones is fed into gates 
and G, respectively which are normally cutoff. 
‘he sampler output at 7 is simply a series of narrow 
kes which occur at a constant repetition period 7’. 
: repetition period is 7’ >> 1/2W where W is the band- 
th of the noise. When G, is gated by FF), the sampling 
ses at 7 are gated through and appear at 5. These are 
lied to G, and G,. The output of G, 1s a sequence of 
ses appearing at the sampling points which correspond 
he values for which x; > Zo. The output at G is a 
ence of pulses appearing at the sampling points which 
espond to the values x; < 2». Hence, the pulses at 4 
6 correspond to the sequence of ones and zeros 
ectively. The output at 4 is applied to terminal S 
‘F,. Each time a one is observed, FF’, is triggered into 
fe one, in turn selecting the forward-count direction of 
reversible binary shift register, RBSR. On the other 
d, when a zero is observed, a pulse applied to F# of 
, selects the reverse-count direction of the RBSR. 
en terminal Ff of RBSR is selected, a pulse correspond- 
to a one is applied to the shift bus. This shifts in the 
nber c into the RBSR. When R is selected, a pulse 
responding to a zero is applied to the input terminal I. 
s shifts in the number one into the RBSR (corre- 
ding to an observed zero). During this time the 
SR remains preset to the number |b|. When the 
SR returns to zero this is sensed at Ho, indicating the 
ptance of H, that noise is present. If it moves up to 
count a + |b|, this situation is sensed at H, indicating 
signal-plus-noise is present. At these instants a pulse 
pplied either to amplifier A) or A, which appears at 
nd resets FF,. This in turn gates out G, and hence 
nd G, essentially disconnecting all input information 
he RBSR. The system remains inactive until the 
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external pulse generator EPG applies a trigger to Gs. 
At this instant FF, is set gating on G,. At the same time 
the gate at 9 is applied to the preset terminal P. This 
presets the RBSR to the count |b|. The system continues 
operating until a pulse is generated at either Hy or H, 
indicating that a decision has been made. The pulse 
repetition period of the EPG is adjusted to be sufficiently 
greater than the time it takes to come to a decision. If 
necessary, the system can be started by triggering the 
EPG manually. Jt is also possible to start a new experi- 
ment by deriving a trigger from the decision output. 

The number of sequential experiments is equal to the 
number of pulses generated by the EPG. This number is 
counted by the preset counter M/, that is, the manual 
set at 13 triggers FF; gating on G;. This permits pulses 
from the EPG to be gated through G;, thus beginning a 
sequential experiment. When M pulses indicating M 
sequential experiments have entered the preset counter, 
FF, is automatically reset at 17, stopping the entire 
system. 

The M counter reads the total number of sequential 
experiments. The a) counter reads the total number of 
times in M experiments that H, was accepted or that the 
reversible counter returned to zero. The N counter reads 
the total number of samples in / experiments while the 
Y(N) counter reads the total number of ones in M experi- 
ments. After M experiments this information is read by 
the observer from the counters. The information is suffi- 
cient to give a single point on the ASN and OC Functions 
directly. 


V. CuHoice or Detector ConsTaANts AND NUMBER OF 
EXPERIMENTS PrER Point 


The total number of sequential experiments per point 
was chosen as M = 100. The values of the detector con- 
stants chosen for the experimental study of the theory are 


(Ue, 

a= —b = 82, (36) 
c= 4, 

c= 8. 


This defines four different physical experiments, for 
example, 
a= 4 
a = 32; c=4 (37) 
8 


a 


a = 32; c 


I 
i 
op) 
ion) 

I 


8. 


Keeping c = 4 and allowing a = 16 and a = 382 is equiv- 
alent to specifying an allowable set of values (po, p,) and 
varying the set a, = 6; to ag = By. From Fig. 2 corre- 
sponding to c = 4 an allowable set of values po, p, is 


i = 0.165, 
pi = 0.240. 


(38) 
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Fig. 2—Graph of p, vs po for two values of c¢. 


Let, 
a = 16. 
The function 
ort mee = 0.0942. (39) 
From (15) 
on mee =i: 
ey zs 
Hence, 
a= 8 = 0.18. (40) 
For the same set of values (po, p:) and a = 32 
a = 8 = 0.0475. (41) 
From Tig. (2) when c = 8, an allowable state is given by 
Po = 0.09, (42) 
pi = 0.135. 
Hence, 
log = a = 0.0507. (43) 
Similar to the previous example, when a = 16 
ae 03 (44) 
When a = 32, 
a = B = 0.156. (45) 


It should be clear that if another allowable pair of values 
(po, Pi) were taken, the corresponding a and 8 would be 
different. 

Because the Wald Theory is only approximate for 
discrete sampling, since the excess over the boundaries 
at the termination of experiment is neglected, it is desirable 
to obtain upper and lower bounds on the OC and ASN 
Functions. These are derived in Wald.’ It can be shown 
that for the constants used, the lower bounds are for all 


practical purposes those given by (25) and (29). For the 
constants the true lower bounds are only slightly low 
(a negligible amount for practical purposes). The upp 
bound for the OC Function is given by Wald,’ 


L’(u) == creak ; (4 
; @ an G 


An upper bound on the ASN Function is also given,’ 
a{l — 2L(u)] cll Eee 

(1 + cp — 1 (1 + oep(u) — 1 

Eq. (48) is well-behaved everywhere except in the neig 

borhood of w = 1. In this region we have the relationsh 

given in (30). 


Ein) = (4 


VI. Discussion or EXPERIMENTAL RESULTS IN THE 
LIGHT OF THE THEORY 


Before attempting to correlate experiment and theor 
some of the unobservable theoretical characteristics w 
be discussed. Fig. 2 is a plot of the allowable paramet 
points corresponding to the parameter values c = 4 ar 
c = 8. It is clear that there exists an infinite number | 
such parameter points. This is consistent with the previo 
discussion. The function c(po, p,) is plotted in Fig. 3. F 
a given value of c, the corresponding set of pairs of valu 
(Po, Pi) can be obtained from this family of functions. 

Fig. 4 is a plot of (22), and Fig. 5 of (25). The importa 
thing to notice is that the functions are monotomic in fl 
strict sense. 

lig. 6 is a theoretical plot of the upper and lower boun 
on the OC Function corresponding to a = |b| = 3 
c = 4. The points represent experimental data. The upp: 
and lower bounds almost overlap each other. Wa. 
predicts for the case where the mean and standard devi 
tion of the measured observable is small, neglecting tl 
excess over the boundaries would have little affect on tl 
OC and ASN Functions. Hence, the experimental poin 
should follow the lower bound. Thus Wald’s work shoy 
that the parameter c must be small in order for the effe 
of neglecting the excess over the boundaries to be sma 
For po = 0.08, p, = 0.375, corresponding to c = 4 
experimental points follow what is essentially the low 
bound. For this case, the lower and upper bounds a 
essentially the same. The small deviations of the cur 
are statistical fluctuations since only 100 experiments a 
performed. This is evident by the manner in which the 
are distributed. Most of the deviations occur in tl 
high slope region for example, in the neighborhood p 
1/(c + 1). This is the region which requires a limit pr 
cess for convergence. 

Fig. 7 is a curve of the corresponding ASN Functio 
Once again the experimental points follow the low 
bound. Note the very close correlation of theory ai 
experiment to the left of the maximum. This is due | 
the fact that for p < 0.18, the hypothesis Hy is accept 


S 


2 
; log (p,/Po) 
C\Po, Py! = 10g (I=po/!=p,) 
p,=.20 
2\ 
8 PP 
s 24 
= 30 
5 40 
= 60 
° 80 
y od 
fo) 
04 08 12 16 
Po 
Fig. 3—Family of curves of c(po, p1). 


p(u) 


u 


4—Universal characteristic of Bernoulli sequential filter in 
region of physical interest. 


. 5—Universal operating characteristic function of sequential 
filter. 


the average approximately 85 per cent of the time. 
snce, the reversible shift register returns to zero 85 
r cent of the time. When Hy, is accepted there is no 
sess over the boundaries at the termination of the 
eriment. That is, the equality sign is realized. There- 
‘e, in this interval, theory and experiment agree very 
sely. On the right side of the peak, the experimental 
nts are very close to the lower bound. However, there 
sts a consistent discrepancy between theory and 
eriment. It is possible that during this phase of the 
instead of the test terminating with H, corresponding 
a value a = 32, it might have terminated at times 
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Fig. 6—Upper and lower bounds on the OC function of Bernoulli- 


sequential filter and the experimental points. 
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Fig. 7—Upper and lower bounds on the ASN function of Bernoulli- 
sequential filter and the experimental points. 


with a = 30 or some number less than 32 although close 
to it. To detect such an error is almost a physical impossi- 
bility since it occurs when the system is operating in its 
most complex manner. That is, it is unlikely that a signal 
of constant repetition period would isolate an error of 
this nature. The error although systematic is still too 
small to be of practical consequence. It is also possible 
that the lower bound is not sufficiently good. It gives 
pessimistic results or results that are always on the safe 
side. This situation is always desirable. 

The points were obtained simultaneously by reading the 
four counters. It is clear from the ASN Function that the 
maximum number of samples taken for a point is at the 
peak or approximately 26,000 samples. This is the most 
critical region. This gives a precision in p of the order 
of 0.6 per cent. The precision in L(p) and E£,(n) is of the 
order of 10 per cent. The statistical characteristics of 
these functions are apparently of such a nature that such 
a precision is sufficiently good. This is indicated by the 
results. It is felt that the simultaneous measurement 
of these characteristics is the reason for good agreement 
with not too large a sample. Fig. 8 is a plot of the OC 
Function for the same values of ¢ but for a = |b| = 16. 
It is clear once again that the experimental points follow 
the lower bound. This indicates that for small values of 
c, Wald’s Theory holds equally well for discrete samp- 
ling. The parameters (a, b) apparently have no influence 
on the characteristics, as predicted by Wald. The trend 
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of the points indicates that the lower bound may be 
slightly high. The difference is however, insignificant. 

The corresponding ASN Function, Fig. 9, also shows 
that the experimental points follow the lower bound. 
The fluctuation in the points is of a statistical nature. — 

By comparing Figs. 6 and 8 it is seen that the effect of 
increasing the numbers (a, b) is to increase the slope of 
the OC Function. This corresponds to a decrease in the 
parameters (a, 8). 

A comparison of Figs. 7 and 9 shows that an increase in 
(a, b) peaks up the ASN Function in the neighborhood of 
p = 1/(c + 1) = 0.20. From these curves it is also seen 
that the ASN Function is a maximum at approximately 
p = 1/(c + 1) = 0.20 as predicted by theory. This is 
the slope of the statistical decision lines. The theory does 
not say that the maximum occurs precisely at p = 
1/(c + 1). It is very close, hence for all practical purposes, 
it can be considered as the point at which the ASN 
Function is a maximum. 

Fig. 10 is a plot of the OC Function for a = b = 82, 
the same as Fig. 6, and for c = 8. It is seen that force = 8, 
the OC Function follows the upper bound. It appears, 
however, that there is a systematic trend for the points 
to be above the upper bound, especially in the region 
0.10 < p < 0.17. The experimental curve is shifted almost 
parallel to the lower bound. This is contrary to the way 
the upper bound behaves. There is no obvious reason for 
experiment and theory to differ in this manner. Intuitively 
we might expect the upper and lower bounds to be parallel 
in the almost linear region as indicated by experiment. 
A look into this matter strictly from a theoretical poimt 
of view might be desirable. From the practical point of 
view, the agreement between theory and experiment is 
sufficiently good. 

Fig. 11 is the corresponding ASN Function. Once 
again it is noted that the curve to the left of the maximum 
agrees with Wald’s Theory, while the part to the right 
follows the upper bound. Most of the points on the right 
are less or equal to the upper bound. The maximum of the 
ASN Function occurs at the point given by theory 
p= 1f/fe+ 1) 0.11. 

Fig. 12 is a plot of the OC Function for a = |b| = 16 
and c¢ = 8. Most of the points lie within the two bounds. 
Once again the trend of the experimental points in the 
almost linear region is to lie on a curve parallel to the 
lower bound. This agrees with Fig. 10. It should be 
emphasized that this is probably inherent in the approxi- 
mation of the upper bounds. Fig. 13 is the ASN Function 
corresponding to a = |b| = 16, c = 8. Once again the 
points follow the upper bound to the right of the maximum. 
To the left of the maximum, the experimental points 
exceed the theoretical bounds. The maximum of this 
curve is very broad. Hence, the neighborhood of the limit 
point p = 1/(e + 1) = 0.11 is large. This implies that 
the Wald bounds are not very good over a wider interval. 
The points corresponding to p > 0.15 do follow the upper 
bound. The validity of the approximation is therefore 
shown to be good in this region. It should be pointed out 
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Fig. 8—Upper and lower bounds on the OC function of Bernoul 


sequential filter and the experimental points. 
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Fig. 9—Upper and lower bounds on the ASN function of Bernoul 
sequential filter and the experimental points. 
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Fig. 10—Upper and lower bounds on the OC function of Bernou! 


sequential filter and the experimental points. 


that a plot of the upper bound in the nieghborhood 
p = 1/(e + 1) will produce a violent oscillation. This 
not shown since it simply was an indication that tl 
bounds are not good in that region. The curve drawn 
that interval is somewhat of an extrapolation. 


WIDE ConcLusions 


In general, the agreement between the Wald Theo 
which neglects the excess over the boundaries at t 
termination of a sequential experiment and the results f 
discrete sampling is good. The upper and lower boun 
on the OC and ASN Functions are also good for practic 
purposes. The discrepancies in general are not of a stat 
tical nature. They are small but systematic and are probak 
due to the fact that the expressions for the bounds al 
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Fig. 11—Upper and lower bounds on the ASN function of Bernoulli- 
sequential filter and the experimental points. 
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Fig. 13—Upper and lower bounds on the ASN function of Bernoulli- 
sequential filter and the experimental points. 


involve certain approximations which are claimed to have 
only a small effect on the results. The experiments bring 
this out. 

The experimental results verify the transformed equa- 
tions in terms of the w variable. It is shown conclusively 
that only the (a, b, c) transformations are significant in 
the sequential observation. This is important since it 
shows which parameters are of practical significance and 
which are only meaningful conceptually. It shows that 
the approximations depend on a functional relationship 
of (Po, Pi, @, B) and not directly on these parameters. 
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The experimental results actually are true for an infinity 
of Bernoulli Sequential experiments which satisfy the 
given (a, b, c) transformations. 


VIII. ADDENDUM 


THe RESONANT PROPERTIES OF A SEQUENTIAL DETECTOR 


One of the most striking phenomena observed during 
the experiments was the behavior of the ASN Function 
in the neighborhood of the indeterminate point p’ = 
1/(c + 1). As seen from the curves, in this region a 
resonance phenomenon is observed. If we use the random 
walk interpretation of sequential detection, then the con- 
cept of resonance in sequential analysis is clear. For 
example, sequential detection can be considered as a 
random walk problem with two absorbtion barriers. 
Thus the particle is trapped between the two barriers 
(a, b) with a probability unity that it will emerge even- 
tually. There is, however, one distribution corresponding 
to a given set of threshold constants (a, b, ¢) which will 
force the particle to oscillate, at random of course, back 
and forth about its zero position (in this case about the 
preset number |b|) and hence spend the maximum time, 
on the average, between the absorption barriers. This 
is the point at which the ASN Function is determined 
by a limit process. The sharpness of this resonance can, 
of course, be made as closely as desired by choosing the 
parameters (po, Pi) or in general (4), 6;) as close to each 
other as desired. The sharpness of this resonance has a 
very striking similarity to the usual resonance phenomenon 
encountered in a linear RLC circuit. If we stretch our 
concepts somewhat, we can say that a sequential detector 
of given constants (a’, b’, c’) is a “statistical resonant 
circuit tuned to the distribution parameter 6 = 6’.”’ Let 
us therefore examine the properties of the sequential 
detector by introducing the concepts inherent in resonant 
circuit theory. This will give further insight into the 
behavior of sequential detectors. 

Let us consider the familiar one parameter family of 
distributions given by 


P(a, 6) = v(0)w(ax)e"* (49) 


which is a special case of the family of distributions given 
in (1). This class includes the familiar and important 
distributions such as the one parameter Gauss, the 
Bernoulli, Poisson, etc. For this special case it can be 
shown that the decision regions are given by 


De x; >a’ +ne’—> Hz, (50) 
ys x; <b’ + ne’ > Ao, (51) 
t=1 
where 
log d ee A 
a’ (52) 


~ q(:) — q(o) ’ 
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log 
bf) = 3 
q(0;) — q( A) (63) 
log v6 ) 
ae) 6,) 
( — Set 54 
q(A,) — (Go) ce 


The ASN Funetion, which is the characteristic of 


interest for this discussion, 1s given by 


b’L (6) + a’[1 — L(6)] 
E(x — c’) (55) 


AGO) = 


At the indeterminate point @ = 6’ corresponding to the 
value of 6 for which H,(x) = c’, we have for the ASN 
lunction, 


a’b’ 
Ey, (« —¢) 


Ey. (n) = (56) 
For the Bernoulli case Ly(v — c')? = p’(1 — p’) and 
with the relationships between (a’, b’, c’) and (a, b, c) 
one can obtain (30). 

It is known that the ASN Function is for all practical 
purposes a maximum at 6 = 6’. At the threshold parameters 
6 = 6; and 6 = 6,, L(@) = 1 — a; L(6,) = B. Hence, 


AO ie 
E,,(« — ¢) 


Ey (n) = 


and 


—Ba’ — b’) +a’ 
E, (a — c’) y 


knowing that H,,(n) < Hy (n) and Hy(n) < E,(n). 
Irom previous considerations we have, 


E,,(n) = (58) 


E(x — c’) = E(x) — Ey/(x) (59) 
and 
Ey(a — c’)’ = Ey {[a — Ey-(x)}?} = o3/(z). (60) 


Thus the maximum of the ASN Function is given by 
(56) which when combined with (60) gives 
a’b’ 
Ey(n) = —>z—7: 
eae 5 
We now define the value of the ASN Function at the 
threshold parameters (65, 6) as, 


(61) 


Ean) = Ey,(n) = kE,(n): exe Ae (62) 


Substituting (57) and (58) into (62) and using equation 
(59), then solving for #y,(x) and H4,(x) yields 


Pei: ; CO ete Oe os 
Ey,(2) = E,(x) — @ nt a5 (x) (63) 
and 
7 ay \ Slay aa b’) ae a’ 2 
Eo, (x) = Ey (x) = are o9/(x). (64) 


For the class of distributions considered, the functions 
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E(x) are monotonic increasing with @ in the strict sense. 
We can now define a bandwidth, 


W = E, (x) — E,p,(x) 


= eae (a’ — b’)o4/(z). (65) 
Since /,(x) is monotonic in 6, we know that for a given 
bandwidth w = 6, — 6) there corresponds a bandwidth 
W = E,,(x) — E,,(z). We can therefore specify the 
bandwidth w by specifying 6, and 6, and this in turn 
determines W which is required in (65). (In certain cases 
such as the Bernoulli case w = W = p, — po.) 

For convenience, let Hy-(n) = N. Then solving (61) 
for b’ and substituting into (65) gives the quadratic 
equation 


, WN BAe : 
(a’) —a’ + No,(x) = 0. (66) 
al 
k 
When (66) is solved for a’ we have 
; WN 
lt ee 
mae a NCE 
[=—ea] 
I WN | sh a 
= 9 (ee 4No;,. (x). (67) 
k 
In order for (67) to make sense, it is necessary that 
A | Ste 
c oun ote ay 
k 
or 
eS ae k o4'(2) 
Gk “e+ a ae ee 


We also know that a’ > 0. The right part of (68) is recog- 
nized as a familiar relationship in normal statistics. It 
can be easily verified that for the case 


k W _ gor(x) 
eS 2 aN (69) 
we have, a’ = —b’ ora = 8. Furthermore, 
k WN ae 
a=) b= ‘as = 9 ) = V/N og (a). (70) 


For this special case, the decision regions are given by 


3 


[x; — E,-(x)] (71) 


i 
N t=1 W ( k ) 


It should be clear that once W is specified in terms of 
6, and 6,, H4,(x) is also specified since H,,(x) is a function 
of @, and 6). Since interest here lies in the resonance phe- 
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omenon, let us rewrite (71) as, 


k W ip = e W k 
Se, |< 


1 20.2 > NS Me 


2 
he 
| 
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Let us assume that it is desired to determine if a random 
variable belongs to a distribution whose parameter is 6’ 
Where 0) < 6’ < 6,. When the unknown parameter is 
; = @' it is known that the number of samples N required 
ior the sequential test to terminate is a maximum. If 
6, — 6) is very small, there is a sharp maximum. We 
herefore specify a number N, such that, if N > No, 
hen it is known with a predetermined probability that 
oe .0 = 6, andat NV < No, then ¢@ <6, or @ > 6,. We 
tan therefore consider the sequential filter when operating 
in this mode as a parameter filter which operates in a 
manner which is clearly analogous to a resonant circuit. 
‘or example, if the input to a resonant circuit is one of a 
set of sinusoidal signals and if it is required to determine 
f a particular frequency is present, then the resonator is 
tuned to this frequency, and the response is measured to 
letermine the presence or absence of a sinusoid at that 
irequency. Similarly in the Sequential Filter described, 
‘tuning’ is accomplished by adjusting the value of 

a(x) for a particular 6’. If the “response” is n > No, 
then a random variable whose distribution has the param- 
i 6 is detected. In this same way, we can use a bank of 
sequential detectors “tuned” over an entire range of 
Jarameters and recognize or estimate one of a set of 
oredetermined parameters. It therefore appears that in 
this mode of operation the sequential detectors can be 
used in multivalued decision problems. The decision to 
accept a hypothesis is made by comparing the responses 
number of samples) at the output of each sequential 
etector after experimentation has terminated and choos- 
ng the hypothesis which corresponds to the largest 
esponse (maximum number of samples). It should be 
rlear that the random variables need not belong to one 
amily but can belong to any member of the exponential 
lass. 

In order to illustrate these ideas more clearly, we will 
onsider (72) in more detail. From (70), we can rewrite 
72) as, 


=) ene 


VN o9 (2) oe 


| i=1 
Let n = kN be the minimum number of samples required 
Lo accept the hypothesis that 0) < 6 < 6, that is, that 
the parameter @ is contained in the confidence interval 
6), 6,). We can use this fact in (73), which gives 


en > [x; — Ey(x)] < 1 : 
Vi 2 VN ala) = Vi 


For kN > 1 (of the order of 100), the random variable, 


(74) 


kN 


© (e: — Be) 
VEN o4/(2) 


Y= (75) 
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is approximately normal with mean zero and standard 
deviation unity. Let 


| 1 
ele (as Nn 
‘ : Vk re Vk 


be the probability that when 6’ is the true parameter, the 
random variable Y will be observed in the designated 
interval. Thus, the constant /& previously defined as 


(76) 


Hy,(n) - Ey s (n) 


ame Neos 


is all that is required to find the probability corresponding 
to a given confidence interval when the number of samples 
is large, for the entire class of distributions considered here. 
It can be shown that corresponding to T = 0.95 we have 
k = 0.25. Thus in designing such a statistical resonant 
filter, we specify the threshold parameters (6, 0,). This 
determines Hy (x) and o» (vx) or W. The constant k 
which defines the confidence interval is then chosen for a 
given confidence probability. Iurthermore, kN is chosen 
sufficiently large so that the large sample distribution of 
the random variable Y is approximately normal. This, 
of course, defines the peak response N and the entire 
sequential filter. In this mode of operation the probability 
[,, is used instead of a, although a specification of either 
determines the other. 

The indicated theory can be used to design a probability 
distribution analyzer which measures the empirical 
distribution of an independent set of samples. This can 
be accomplished by defining a set of slicing thresholds on 
the random signal such that the output of the 7th slicer is, 


GU.) === ale fT, 


(77) 
= 0 otherwise. 
The slicer output is a set of zeros and ones distributed in a 
Bernoulli Distribution of probabilities (p;, 1 — p,). It is 
required to estimate the p(x,;) for each slicing threshold. 
For the Bernoulli case, 


Ey(z)=p', ova) = Vp-p). (78) 
For a typical point p’ on the distribution and for a 95 
per cent confidence interval, (74) becomes 
ee ee (79) 
Sete ND) eng ie 

In order to find the value of p’ we “tune” the filter by 
varying p’ until the “response” is n > kN. When this 
occurs, we have found the required value of probability 
corresponding to the particular slicing threshold. The 
standard way of performing this measurement is to count 
the total number of samples and the number of ones. The 
frequency at the threshold is obtained by taking the ratio 
of ones to total number of samples. The latter method 
can be considered as analogous to measuring the frequency 
of a sinusoid by counting rather than measuring the 
response of a resonant circuit. 


= il 
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Minimum-Scan Pattern Recognition’ 
ARTHUR GILLt 


Summary—Speedier and simpler pattern-recognition systems 
can be realized when provided with a minimum-scan reading de- 
vice. For the idealized case, where the input set of patterns is finite, 
binary and errorless, a theorem is proved which enables the de- 
signer to predict the efficiency range of the contemplated reading 
device. A constructive method, which can be readily programmed 
for computer processing, is proposed for finding the shortest scan- 
ning path realizable for any given set. In the case of noise, scan- 
ning paths are sought which maintain a prescribed minimal “‘dis- 
tance’’ between the patterns, and hence yield a prescribed level of 
error-detecting capability. The theorem previously proved is ex- 
tended for this case, and a constructive method is proposed for 
finding the shortest path consistent with any specified minimal 
distance, for any given set of patterns. 


INTRODUCTION 


HE problem of automatic processing of printed 
ale documents has been receiving an increasing amount 

of study in recent years. Various character-reading 
systems have been designed, and a number of papers have 
been published with special emphasis on the mechanical 
and electronic aspects of the pattern-scanning mechanism, 
or on the judicious coding of the read-in material. In 
virtually all the proposed systems the reader—the device 
which transfers the given pattern information to the 
coder—is designed to scan each pattern completely, 
without any regard to the redundancies which may be 
introduced. The entire task of removing the redundancies 
is imposed on the coder which, following some statistical 
criteria, converts the scanned information into code- 
words suitable for fast and reliable recognition. 

In this paper, the problem of designing a more “‘intelli- 
gent”? reader is considered. Such a reader will not scan 
the entire area of each given pattern, but a judiciously- 
chosen portion of it. The result in many practical cases 
is a considerably speedier operation, as well as simplifica- 
tion in the coding and recognizing parts of the system. 
The scanned portion of each pattern—or the scanning 
path—is designed to be the same for all possible input 
characters; consequently the reader is required to have 
a fixed mechanism to make it ignore a predetermined 
portion of the pattern area. Thus the higher speed and 
simplification may, in terms of hardware, be achieved 
quite inexpensively. 

The paper is restricted to preliminary analysis where 
the following assumptions are made: 1) the number p of 
different patterns which may be fed into the recognizing 
system is finite; 2) all patterns can be divided into a finite 
number of cells c, each of which is completely black or 


* Manuscript received by the PGIT, August 1, 1958. The re- 
search in this paper was supported by the U. 8. Navy, under con- 
tract with the University of California, Berkeley. 
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completely white; 3) a perfect positioning mechanism is 
available. 

Under these assumptions, the set of different input 
patterns can be conveniently represented by a p X ¢€ 
matrix of binary digits, where unities and zeros represent 
black and white cells respectively. Thus, in such a matrix, 
the element common to row 7 and column 7 will describe 
the j’th cell of the 2’th pattern. Matrix (1) below is an 
example of a pattern matrix with p = 6 and ¢ = 9; the 
indexes attached to the rows and columns are the serial 
numbers of the patterns and cells respectively. 


A Pattern Matrix 
(1) 
123456789 
111001111 | 
111001001 
1190100101 
101001101 
001110100 
000101110 | 


fwd 


> Or 


ParreRN Martrrrx REDUCTION 


For the idealized input described above and under the 
assumption of errorless reading, the most efficient reader 
would be the one which can distinguish between the 
various patterns while scanning the minimum number of 
cells. Thus, our first task in designing the reader is to 
find the minimal scanning path realizable for the given 
set of patterns. In terms of the pattern matrix, this task 
corresponds to finding the largest number of columns 
which can be deleted so that the rema‘ning rows will 
still be distinct. The result of this reduction operation is 
a minimal pattern matrix which, out of all irreducible 
pattern matrices, has the smallest number of columns. 
Before describing any minimal reduction schemes, it is 
useful to state and prove the following theorem: 

If p X c 7s the dimension of the original pattern matrix 

and Pp X Cmin the dimension of the corresponding minimal 

matrix, then: 


{log py < Ga SS pol 


where the notation {x} represents the smallest integer 
larger than or equal to 2. 
Proof 

The total number of different patterns constructable 
from Cmin Cells is 2°"'*; hence: 


p << Ores 


Or: 


IV 


logs p. 


Cmin 


mce Cyin has to be an integer, 
uivalent to: 


the last inequality is 


{logs p} 


Cmin = 


ich verifies the lower limit in the theorem. 

To verify the upper limit, consider the original p X ¢ 
jatrix in which the rows have been arranged in a sequence 
decreasing numerical value (each row being regarded 
}a binary number). Now proceed from left to right and 
lark the column in which row 1 and row 2 first disagree. 
he, with rows 2 and 8, 3 and 4, down to p — 1 and 
Deleting all unmarked eolane leaves a matrix which 
48 no more than p — 1 columns. To show that in the 
duced matrix no two rows are identical, consider any 
ur of consecutive rows, say 7 and 7 + 1, and suppose 
jat they first disagree in column j. These rows are then 
entical to the left of 7; in column j row 7 is unity and 
yw 2 + 1 is zero. Consequently, in the reduced matrix 
s well as in the original matrix, row 7 has to be numerically 
rger than row 7 + 1. Since this argument applies to all 
Airs of consecutive rows, each row has to be numerically 
maller than the preceding one, and hence no two rows 
an be alike. Thus, it is always possible to reduce any 
ttern matrix to one containing no more than p — 1 
plumns. This reduction scheme, carried out on the set 
atrix (1)], is demonstrated in matrix (2); matrix (3) 
the resulting reduced matrix. 


Nonminimal Reduction of Matrix (1) 


(2) (3) 
| Bll? 3 4 556.7.8.9) VY 
eed ad O10 al (ia 4 1) qi 
2 1 a 00 Ke 01 oy Tel) 
2 ie O10 0 0M 3] 1101 
4 S Git-0 OM ONt 4/1011 
51 \o A 111,004.20. 5 | 0011 
CA OVO LTO. 160s E20; 6 |.0001 | 
vat Ff 


| To verify that the upper limit may be reached, one has 

construct a p X (p — 1) irreducible pattern matrix. 
his can be readily done by forcing row 2 to be identical 
row | in all columns but column 1, forcing row 3 to be 
entical to row 1 or 2 in all columns but column 2, and 
general forcing row 7 (¢ = 2, 3, --- p) to be identical to 
ny of the preceding rows except in column 7 — 1. The 
sult is a matrix in which all the rows are distinct and 
1 which two rows become identical if any one of the p — 1 
olumns is deleted. Thus the proof of the theorem is 
pmplete. 


READER EFFICIENCY 


For the set of p patterns in which the 7’th pattern has 
ae probability P,, the average information rate is given by: 


ia SOU NP: logs 


P; bits/pattern. 
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If each pattern is represented by c binary digits, the trans- 
mission rate is given by: 

T = c binary digits/pattern. 
The redundancy p in the system can be defined by: 


ESR 
ew oy) 


and the efficiency of the system by: 


i=1 


Using the theorem proved in the preceding section, we 
can write: 


= 5 P; los P; — >> P; log, P 
a=1 t= 
Dp aad {loge p} 


This inequality yields the range of efficiencies which may 
be realized with a reader employing a minimal scanning 
path. For 10 equiprobable characters, for example, the 
efficiency of a minimal-scan reader will always le between 
0.37 and 0.83. Knowing the initial value for c, such informa- 
tion is invariably useful in estimating the advantage 
gained by designing such a reader. If in the above example, 
the character style requires a grid of 50 cells with the 
corresponding efficiency of 0.056, this advantage is 
apparent. 

When ¢ is considerably larger than p, sufficient advan- 
tage may be gained by realizing the lower-efficiency limit. 
When this is the case, the desired path (or paths) can be 
found either by the method described in the preceding 
section, or by a partitioning scheme described in another 
paper.’ Either method is sufficiently simple to handle 
most practical cases without any need for machine com- 
putation. However, since these methods do not guarantee 
that the reduced path will contain less than p — 1 cells, 
they are seldom useful in cases where c is in the same order 
of magnitude as p or perhaps even smaller than p. In 
such cases, more complex schemes have to be employed 
which are capable of producing the minimal path for the 
given set. 


<n 


A Mintmau Repucrion Mrersop 


Consider an arbitrary pattern matrix, and suppose that 
only the pattern represented by the first row is to be 
recognized, in which case any of the cells 1 to ¢ may be 
selected as a legitimate path. Suppose now that pattern 
2 is added to the set. Then each of the paths selected 
previously may or may not be still legitimate, depending 
on whether the part of pattern 2 in that path does or does 
not coincide with the part of pattern 1 in the same path. 
If such a coincidence occurs, the path in question has to 
be augmented with another cell which is numerically 


LA, Glovazky, “Determination of redundancies in a set of 
patterns,” IRE Trans. on Inrormation TuHerory, vol. IT-2, pp. 
151-153; December, 1956. 


M4 IRE TRANSACTIONS ON INFORMATION THEORY 


different in the two coincident patterns. In terms of the 
matrix, if at a certain path rows 1 and 2 coincide, the 
augmenting cell is represented by any column whose 
first and second digits differ. After carrying out all possible 
augmentations, a revised list of paths is obtained including 
all paths which may be used to recognize patterns | and 2. 
The next step is similar to the previous one: we add pattern 
3 to the set and examine the adequacy of each path pro- 
duced previously; if for a given path, row 3 coincides 
with any preceding row, this path is augmented with any 
cell represented by a column in which the two coincident 
rows differ. Carrying this procedure down to the last row 
results in a final list of paths which contains all paths 
adequate for the recognition of the p given patterns. 

It should be noticed that if at any step of the procedure 
a path is found inadequate, its augmentation is always 
possible; this follows from the assumption that no two 
rows in the matrix can be identical in all their digits. 
Also, no more than two rows—one of which is the newly 
added row—can become coincident in any listed path; 
this follows from the fact that any path in the current list 
is by construction adequate for all rows previously 
covered. 

A helpful, though not essential, operation is to eliminate 
from the list produced at the end of each stage all the 
implying paths. An implying path is a path containing a 
group of cells which constitutes another path in the same 
list. This operation not only contracts the size of the paths 
list at each stage, but also restricts it to paths which are 
irreducible; this follows from the fact that, since the paths 
lists are exhaustive, every path which is reducible is 
necessarily implying. 

The testing and augmentation of scanning paths can 
best be done with the aid of the sum matrix, the application 
of which to minimal reduction of pattern matrices was 
first proposed by McCluskey.” Every row in this matrix 
is the sum modulo 2 of two rows in the pattern matrix. 
Consequently, a unity or a zero appearing in any column 
of a sum row, indicate that the component rows differ or 
agree, respectively, in that column. For our purpose we 
construct p — 1 sum matrices, of which the k’th one 
contains the sums of row & + 1 with all preceding k rows. 
Thus, to determine whether, in a given path, row k + 1 
coincides with any preceding row, we test whether any 
row in the kth sum matrix has zeros in all the columns rep- 
resenting this path. If such a row exists, then an augmenting 
cell is a cell represented by any column in which the row 
is unity. 

The set of sum matrices appropriate for the pattern 
matrix (1) is shown below. The indexes attached to the 
rows indicate the component rows in matrix (1). To test, 
for example, the adequacy of the path 3-7 listed at the 
end of the second cycle of the procedure, we examine 
all rows in sum matrix No. 3. Row 4/1 of this matrix has 


2. C. Riekeman, A. Glovazky, and E. J. McClusky, ‘“De- 
termination of redundancies in a set of patterns,’ IRE Trans. on 
INrorMATION THEORY, vol. IT-3, pp. 167-168; June, 1957. 
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zeros in both columns 3 and 7, which indicates that the 
path has to be augmented. Augmenting cells are either 
2 or 8, since the corresponding columns have unities at 
row 4/1. The path 3-7, therefore, yields the augmented 
paths 2-3-7 and 3-7-8. 

The following is a summary of the proposed minimal 
reduction scheme, in a form suitable for computer pro- 
gramming: 

Initial paths list: 1,273.35 ereec 
Initial value for k: 1. 

1) For each path in the list produced in the previous 
cycle, determine whether the corresponding group 
of columns in the k’th sum matrix has a zero row. 

2) If there is such a row, augment the path with a cell 
corresponding to each column in the sum matrix in 
which the row is unity. 


Sum Matrices for Matrix (1) 


(4) 

123456789 

No. 1 2/1 [000000110] 
No.2  3/2{ 001101100 | 
3/1 |001101010 | 
4/3 | 011101000 | 

No.3 4/2 | 010000100 
4/1 | 010000010 | 
5/4 | 100111001 | 

No.4 5/3 | 111010001 
5/2} 110111101 
5/1 {110111011 | 
6/5 | 001011010 | 

6/4) 101100011 

No.5 6/3} 110001011 
6/2} 111100111 
6/1 |111100001 | 


3) From the expanded path list, eliminate all implying 
paths. 

4) lik < p — 1, add 1 to & and return to step 1). If 
keep 1, the list would contain all irreducible 
paths which are adequate for recognizing the p 
given patterns. The shortest paths in this list are 
the desired minimal paths. 

The table shown demonstrates the procedure for reduc- 
ing set (1), utilizing the sum matrices (4). The crossed-out 
paths are the implying paths eliminated at the end of each 
cycle. The resulting minimal matrices, corresponding to 
the shortest paths 2-4-8 and 2-6-8, are given in matrices 
(5) and (6). 

As can be noticed, a nonimplying path in the list pro- 
duced at the end of the first cycle cannot contain more 
than a single cell. This can be explained by the fact that 
it is always possible to find a single column in which 
rows | and 2 of the pattern matrix differ. Also, as shown 
above, no path requires more than one additional cell 


Minimal Reduction of Matrix (1) 
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* Cycle | Cycle 2 Cycle 3 Cycle 4 Cycle 5 
Ni- : ‘as = = i 
tial Aug- Aug- Aug- Aug- Aug- 
aths | menting Paths menting Paths menting Paths menting Paths menting Paths 
list cells list cells list cells list cells list cells list 
L 78 te 37 28 237 14569 1237 none 1237 
2 78 —+8- 47 28 378 14569 ae 2357 
3 78 2 67 28 247 none 2357 none 2379 
4 78 —28- 78 2346 478 12359 —236Ta 1378 
5 78 = Shae 38 Pal 267 none 2379 none 3578 
6 78 38- 48 Diy 678 12359 1378 none 3678 
7 none —+i—- 68 27 278 14569 347 3789 
8 none 8- F8- 378- 3578 none 2347 
9 78 5T— -478- 3678 none 2457 
ie bF8- 3789 none 2467 
—+47- 238 14569 247 3568 2478- 
68— 38 1478 none 1478 
7 3468 248 none 2478- 3478 
8 3467 -478- 3478 none || 4578 
T5- 268 none 4578 none A789 
—89- 678- 3789 none 1267 
267 13489 2367 
1678 none 2467—- 
3678- 2679 
5678 12349 1678 
6789 none 15678- 
1278 none 25678-— 
_35678- 
2578 none A5678— 
56789 
2789 none 6789 
1238 none 1278 
2348- 2578 
2358 none 2789 
1238 
2389 none 2358 
248 none 2389 
| 268 none 248 
268 


Minimal Forms of Matrix (1) 

(5) (6) 

[248] [268] 

iL 1@il Ih TE 
2 100 ZALO 
3 110 3 100 
4 000 4 010 
5 010 5 000 
6 O11 6 O11 

er cycle of the reduction procedure. These observations 


nd the fact that the procedure terminates after p — 1 
ycles again verify that no minimal path can contain 
nore than p — 1 cells. 


ERror-DETECTING SCANNING PATHS 


The minimal-reduction scheme described in the preced- 
ng section guarantees that the “distance” between any 
wo patterns, 7.e., the number of differing cells, will be 
t least 1 in the determined paths. When the input 
atterns are not distorted and the scanner is ideal, this 
uarantee is sufficient to insure unique identification of 
ll the given patterns. In the “noisy” case, however, 
vhen either the patterns or the scanners are imperfect, 
is desirable to maintain a larger distance between any 
wo patterns, so that one or more errors may be detected 
r corrected. If the distance maintained between any two 


patterns is at least d, then it is always possible to detect 
d — 1 errors or correct {d/2} — 1 errors in any given 
pattern.” 

Our purpose in the noisy case is then to find scanning 
paths which are consistent with a prescribed minimum 
distance d. Such paths, which can always be found for 
d = 1, are not always realizable for d > 1. Certainly, 
no path consistent with a specified d can be found if the 
minimum distance d, in the original pattern matrix is 
smaller than d; in such cases, therefore, no “noise-proofing”’ 
is possible unless an increase in ¢ is permissible. In the 
following discussion we shall assume that d < d, and that 
c is fixed. 

A scanning path consistent with a prescribed d can be 
found-—if it exists—by the following scheme. Arrange the 
rows of the pattern matrix in a sequence of decreasing 
numerical value and find a reduced matrix by the method 
demonstrated in matrices (2) and (3). Now, ignoring 
rows which are already d cells away from all other rows, 
arrange the remaining (unreduced) part of the original 
matrix in a descending sequence and reduce it by the 
same method. Combine the two reduced matrices and 
again, ignoring all rows which are already d cells away 
from all other rows, reduce the remainder. Continue this 
process until all rows are at least d cells apart from each 


3R. W. Hamming, ‘Error detecting and error correcting codes,” 


Bell Sys. Tech. J., vol. 29, pp. 147-160; April, 1950. 


o6 IRE TRANSACTIONS ON INFORMATION THEORY 


other. The columns of the matrix constructed by joining 
all the reduced matrices represent, then, a path consistent 
with the specified d. The important fact to observe here 
is that this procedure cannot involve more than d cycles, 
since each additional cycle increases all inter-pattern 
distances by at least 1 (except when a distance is already 
d). Using the theorem stating that no reduced matrix 
can contain more than p — 1 columns, we can now con- 
elude that the minimal path consistent with a specified 
d can never contain more than d(p — 1) cells. To verify 
that this limit may be reached, we can construct d matrices 
of the type described at the end of the second section; 
joining these matrices end-to-end results in a single 
p X d(p — 1) matrix in which the rows cannot be d cells 
apart unless all d(p — 1) columns are maintained. In 
conclusion we can therefore write: 


Cmin =< d(p nas 1) 


The lower limit of c¢,;, 1s dictated by the minimum 
number of cells capable of accommodating p patterns 
with minimal inter-pattern distance d. From recently 
published results,” this limit is given by: 


Cmin Pa | logs p} ar d ia Ih 


Thus we have the general relationship 


{ log. p} aia d es it = Cmin < d(p ae 1) 


which checks with the separately-obtained result for 
d = 1. 

The reduction scheme described in this section may be 
useful when ¢ is considerably larger than d(p — 1). In 
many practical cases, however, c 1s in the order of d(p — 1) 
and often smaller than d(p — 1), and a method of minimal 
reduction is desired. Such a method, revealing all minimal 
paths consistent with a specified d, is described in the 
following section. 


Minima Repucrion Wire PRESCRIBED d 


Consider a sum matrix which contains the sums of all 
possible pairs of rows appearing in the original pattern 
matrix. In terms of this sum matrix, the problem of finding 
the minimal path with a prescribed d can be formulated 
as follows: we wish to delete the largest number of columns 
from the sum matrix so that no row will contain less than 
d unities. The procedure which effects this reduction is 
the following. Consider an arbitrary row in the sum 
matrix, assumedly containing unities in columns 
Ci Gr , €,. Then the minimal matrix has to contain 
at least column ¢,, or ¢o, , Or C,, Since otherwise the 
row in question will appear as a zero row in the reduced 
sum matrix—which is prohibited for any d > 0. Suppose 
now that c, is the column which appears in the minimal 
matrix; delete this column and mark that all rows containing 
unities in ¢, are partially “taken care of,” in the sense 


4 DP). D. Joshi, “A note on the upper bounds for minimal-distance 
codes,”’ Information and Control, vol. 1, no. 3, pp. 289-295; Sep- 
tember, 1958. 
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that now only d — 1 unities have to be guaranteed in 
each one of them. Carry out this operation with respect 
to each of the a columns, thus obtaining a reduced matrices 
at the end of the first cycle. With respect to each of the 
reduced matrices, carry out the same operation carried 
out on the original sum matrix; specifically: consider an 
arbitrary row, delete—-one at a time—columns in which 
this row is unity, and mark the number of unities which 
have still to be ‘‘taken care of” in each row. The result is 
again an enlarged set of matrices which can be processed 
in the same manner. If, at the end of a cycle, it is observed 
that any row has been completely “taken care of’— 
z.e., that in the columns already deleted from the corre- 
sponding matrix this row already has d unities—the row 
may be deleted. The entire process ends when a matrix is 
first obtained in which all remaining rows may be deleted; 
the columns which are absent from this matrix represent 
the cells of the desired minimal path. Since no row is 
omitted from a matrix before its portion in the deleted 
columns contains d unities, the resulting path indeed 
features a minimal interpattern distance of d. It can be 
seen that the various groups of deleted columns which 
are obtained at the end of each reduction cycle must 
include the group contained in the minimal matrix, 
since otherwise a zero row would appear in the minimal 
sum matrix. Consequently, the cycle producing the first — 
minimal path is also the cycle producing all minimal paths. 
A step which is not essential, but in most cases helpful, 
is to select the row which contains the smallest number of 
unities as the ‘arbitrary’ row referred to above. This 
step will reduce to the minimum the number of matrices 
which have to be considered at each cycle, and will often 
accelerate the reduction process. Another helpful step is 
to eliminate all matrices whose rows and columns appear 
in other matrices. Other procedural details can be best 
explained by means of an example. . 
Matrix (7) contains all the rows of the matrices (4), 
arbitrarily numbered, and hence is the complete sum 
matrix of the pattern set (1). The ‘“d,”” number attached 
to each row indicates how many unities in the correspond- 
ing row still have to be “taken care of.” At the outset, 
of course, all these numbers are d which, in our example, 
was chosen to be 2. Row 1 (marked with an arrow), which 
contains the least number of unities, is seen to have 
unities in columns 7 and 8 of matrix (7). When column 
8 is deleted, matrix (8) results, while when column 7 is 
deleted, matrix (9) results. The auxiliary column, now 
called ‘‘d,,’’ 1s modified to allow for the unities which 
already have been “‘taken care of” by the column deletion. 
Thus, a d, number attached to a row which contained a 
unity at the deleted column, equals the corresponding do 
number diminished by 1; all the other d, numbers are 
the same as the corresponding dy numbers. Consequently, 
the entire d, column can be obtained simply by subtracting, 
term by term, the deleted column from the d, column. 
Proceeding with the reduction, the rows to be considered 
in the new matrices are again the ones containing the 
least number of unities—namely row 1 in matrix (8) and 
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Reduction of Matrix (1) with d = 2 


The sum matrix Cycle 1 Cycle 2 
a) (8) (9) (10) 
123456789 do 12345679 d, 12345689 dy 1234569 dz 
—1 { 0000001107) 2 —1 [000000107] 1 —1 { 000000107] 1 1 | 0000000] 0 

2 | 001101010 | 2 2 | 00110100 | 1 2 | 00110110 | 2 2 | 0011010 | 1 

3 | 010000010 | 2 3 | 01000000 | 1 3 | 01000010 | 2 —3 | 0100000 | 1 

4 | 110111011 | 2 4 11011101 | 1 4 {| 11011111 | 2 4 | 1101111 | 1 

5 | 111100001 | 2 5 | 11110001 | 2 5 | 11110001 | 2 5 | 1111001 | 2 

6 | 001101100 | 2 6 | 00110110 | 2 6 | 00110100 | 1 6 | 0011010 | 1 

7 | 010000100 | 2 7 | 01000010 | 2 7 | 01000000 | 1 7 | 0100000 | 1 

8 | 110111101 | 2 Sail OR AS 2, 8 | 11011101 | 1 8 | 1101111 | 1 

9 | 111100111 | 2 Oe OO MS al Oe UI OOM at 9 | 1111001 | 0 

10 | 011101000 | 2 10 | 01110100 | 2 10 | 01110100 | 2 10 | 0111010 | 2 

11 | 111010001 | 2 11} 11101001 | 2 11 | 11101001 | 2 11 | 1110101 | 2 

12 | 110001011 | 2 12 | 11000101 | 1 Pe LODO eZ 12 | 1100011 | 1 

13 | 100111001 | 2 13 | 10011101 | 2 13 | 10011101 | 2 13 | 1001111 | 2 

14 | 101100011 | 2 14 | 10110001 | 1 14 | 10110011 | 2 14 | 1011001 | 1 

15 001011010 _| 2 15 |.00101100_| 1 15 |.00101110_} 2 15 [0010110 | 1 

Cycle 3 Cycle 4 Cycle 5 
(11) (12) (13) (14) (15) 
134569 ds 14569 dy 13569 dy 18459 ds 
—2 [01101079 1 2 701010] 0 2 [01010] 0 2 [01100] 0 4569 ds 

3 | 000000 | 0 5 | 11001 | 0 5 | 11001 | 0 =x |] LMMM@IE | At Sei |i ilalihy at 

4 | 101111 | 0 6 | 01010 | 0 6 | 01010 | 0 6 | 01100; O 

5 | 111001 | 1 10 | 01010 | 0 10 | 01010 | 0 10 | 01100 | 0 (16) 

6 | 011010 | 1 11 | 10101 | 0 1 ELMS eal LS UL Oe a 

7 | 000000 | 0 ale) | Ia || LSP LOMA ee 139) LOUIS el 1569 ds 

& | 101111 | 0 14 | 11001 | 0 14 | 11001 | 0 14 PEO —13 [1111] 1 

10 | 011010 | 1 15 |.00110} 0 sally [ECONUTETCOI) aL 16 |01010_) 0 

11 | 110101 | 1 z (17) 

12 | 100011 | 0 (20) (21) (22) 

13 | 101111 | 2 1469 ds 
14 | 111001 | 1 1669 ds 1369 ds 1359 ds als UY) a 
15 _010110_| 1 LIS LOTS SO ie LL OL EO Uk |My) i 

aol) | VEE OE 13 ou | 0 13 ‘iou| 0 (18) 
_ 15 0110] 0 15 [0110 15 0110] 0 
1459 ds 
(23) (24) (25) (26) 13 [1111] 1 
3459 ds 1459 ds 1359 ds 1345 ds (19) 

IOS 4 (UO) @ 1101) 0 1110 
Zee ON 0 Ht || OIL @ aa ae 1101 1456 ds 
. 13 | 0111] 0 15) allan at 1011 0 1011 —13 [1111] 1 

14 L1101_) 0 1A ANON 1101 1110 


Minimal Forms of Matrix (1) with d = 2 


(27) (28) (29) 

24578 12678 26789 
1{ 10011 ie C1111 ieee 
2 | 10000 2| 11100 2| 11001 
3 | 11010 3 | 11010 3 | 10101 
4 | 00010 4| 10110 4} 01101 
5 | 01110 5 | 00010 5 | 00100 
6 01011 6 00111 6 01110 


ow 1 in matrix (9). These, in turn, call for the deletion 
f columns 7 and 8 from matrix (8) and matrix (9), 
espectively. The resulting reduced matrices happen to 
e identical, and are represented by matrix (10). Producing 
he auxiliary column d, from d, in the same manner that 
, was produced from do, it is seen that the d, numbers 
or rows 1 and 9 of matrix (10) are zero. It can be con- 
luded that these rows are completely “taken care of” 
nd hence may be deleted from the matrix. Subsequent 
ycles are carried out in the same manner until, as in the 
th cycle of our example, matrices are obtained with an 
uxiliary column which is composed of zeros only. Such 
atrices in the example are (21), (23) and (26). The 


groups of columns which are absent from these matrices 
represent the sought scanning paths, guaranteeing at 
least one error-detection capability. Matrices (27)—(29) 
are the corresponding minimal pattern matrices. 

The following summarizes the procedure, again in a 
form suitable for computer programming: 

Initial set of matrices: the complete sum matrix. 

The dy column: d for all rows. 

Initial value for k: 1. 

1) For each matrix in the set produced in the preceding 
cycle: a) select the row which contains the least 
number of unities; b) produce new matrices by 
deleting, one at a time, columns in which the selected 
row is unity; c) for each matrix thus produced, 
determine the d, column by subtracting from the 
d,. column of the originating matrix the column 
which was deleted. 

From the new set of matrices, eliminate all duplicate 
matrices. 

Test whether any d, column consists entirely of 
zeros. If such a column exists, the columns which are 
absent from the corresponding matrix represent the 
desired minimal path. 


3 


— 
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1) If the test 
numbers are zero; add 1 to k and return to step 1). 


3) is negative, delete all rows whose d, 


This procedure may, of course, be used in the special case 
d = 1, for which another procedure has already been 
described. The advantage of the other procedure is that 
all irreducible paths—minimal and nonminimal—are 
produced simultaneously, which may be desirable from 
the designer’s point of view. The disadvantage is that the 


method cannot be readily extended to cases where d > 1. 


CONCLUSION 


The theorems and algorithms developed in the preceding 
sections are valuable in the design of a minimum-scan 
pattern-recognizer which is to operate with a fixed and 
highly-standardized set of input patterns. A familar 


June 


example is a reading system for standard alpha-numerical 
characters, the potentialities of which in the field of 
commercial communication are apparent. 

It should be pointed out that although the material 
presented above is concerned with visual patterns only, 
the results obtained may be directly applied to any finite 
set of binary messages. The problems encountered in the 
design of an identifying device for such messages is exactly 
the same as those encountered in designing a ‘“‘reader’’ 
for pattern-recognition systems. 
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On the Mean-Square Noise Power of an Optimun Linear 
Digital Filter for Correlated Noise Input’ 


MARVIN BLUMT 


Summary—An asymptotic solution for the mean-square output 
noise power of an optimum digital filter is obtained. It is assumed 
that the input consists of a polynomial plus correlated noise. The 
asymptotic solution is found by fixing the interval between samples 
and allowing the number of samples to approach infinity. The 
solution obtained for the minimum variance filter is compared with 
the solution as obtained for the ‘‘least-squares filter,’ and -it is 
shown that the latter filter is asymptotically efficient as compared 
to the former. It is shown for each of the above filters that the 
mean-square output noise power is proportional to the spectral 
density function of the correlated noise, evaluated at zero fre- 
quency, and that the factor of proportionality is the same. 


HIS paper presents an asymptotic solution for 

evaluating the output mean-square noise power of 

an optimum linear digital filter when the imput 
consists of a polynomial of degree n plus correlated noise. 
The asymptotic solution is obtained by fixing the interval 
between samples (7’) and letting the finite memory of the 
filter (m7) approach an infinite memory filter as the 
number of input samples (m + 1) is allowed to approach 
infinity. 

Two classes of input functions are considered as follows: 
a) all polynomials of degree n plus uncorrelated noise of 
mean square given by ay; b) all polynomials of degree n 
plus correlated noise of mean square given by cy. 


* Manuscript received by the PGIT, August 28, 1958. 
+ Convair-Astronautics, San Diego, Calif. 


Two optimum digital filters are discussed as follows: 
A) a least-squares linear digital filter which is optimum 
with respect to the class of input functions a; B) a mini- 
mum variance linear digital filter which is optimum with 
respect to the class of input functions ). 

Three combinations of input functions (lower case) and 
filters (upper case), and the corresponding output mean 
square noise powers at time m7’ were investigated. They 
are: 

1) Case aA with output mean square given by o, 

(see Blum’) 

2) Case bB with output mean square given by ¢ 

3) Case bA with output mean square given by ¢;. 

The following theorems are proven: 


niw 


I. If N(¢) is a random process whose spectral density 
function f(A) is plecewise continuous and positive, then 


iim ==) — ale 

mo Om 
This theorem shows that the Jeast-squares filter is asymp- 
totically efficient as compared to the minimum variance 
filter. 


'M. Blum, “On the mean-square noise power of an optimum 
linear discrete filter operating on polynomial plus white noise,” 
IRE Trans. oN INFORMATION TuEory, vol. IT-3, pp. 225-231; 
December, 1957. 
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IJ. The asymptotic value for ¢% is given by 


G,, Qa 
Lim zp it f(O). 


vet the input to a digital filter be sampled from the function 


e(t) = P(t) + N(d) (1) 


There 


I 


P(t) > a,P(t) 


= 0 


0 (2) 


20 


Sa nonrandom regression function whose components” 
.(t), k = 0, 1, 2, --- mn are known, but whose parameter 
rector @ = (a, --- a,) is not known, and MN (f) is a sta- 
jionary random noise process whose ensemble average is 
cero for all ¢ and whose correlation function is given by” 


E{N(@N(s)} = one(t— 8), (0) = 1. (3) 


Petne a linear digital filter’ with weighting sequence W.,,. 
Then the output at ¢ = mis given by,’ (m = 0, 1, 2, ---) 


ec. ae 2 Wea (4) 
u=0 


where W, = Ortoru <0 and @ > om: 
_ The output at time ¢ = m is an estimate of a desired 
(a 


Sa = i : Meelis ae (5) 


b 

where k(r) is defined by the desired linear operation on 
the input information P(t). Then the optimum weighting 
equence W in the minimum variance sense, 7.¢., 


Etex — S*} = 0 
Eiet — Sk}? = 6,= 


a minimum, (6) 
s given by the matrix equation’ 

Wee VBR VESPA). (7) 
where V isthe (m + 1) X (m + 1) symmetric correlation 


matrix with elements in the 7th row and jth column given 


aw 2) gale 2 an 1 (8) 


| 
| 
| 2 The equations for the weighting sequences (7), (11) and mean 
|quare output ratios (15), (16), and (17) are valid for a general 
‘lass of regression function. In order to prove the theorems, the 
P(t) will be taken as orthogonal polynomials with respect to sum- 
ation at the sampled points. 
3 The operator H{ } denotes the ensemble average of the brack- 
sted quantities. 
| 4H. M. James, N. B. Nichols, and R. 8. Phillips, M.I.T. Rad. 
Lab. Series, McGraw-Hill Book Co., Inc., N. Y., vol. 25; 1947. 
5 Note that the sampling interval 7’ will be taken as unity to 
simplify the notation. The effects of 7 # 1 will be considered 
ater. 
6 The notation: —1 indicates an inverse matrix, and a prime the 
ranspose matrix. 

7M. Blum, “An extension of the minimum mean square pre- 
iction theory for sampled input signals,’ IRE Trans. on INror- 
ATION TueEory, vol. IT-2, pp. 176-184; September, 1956. 
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P is the (n + 1) X (m + 1) matrix whose element in the 
ith row and jth column is 


eee 
ere 


of ae ll (9) 
7m = 1, 


and Q is the column vector given by Blum’s (11)’ 


P;-i[m = (j i, 1)] 
j = 


oo 


[Pim = Dar, j= 


Q; = Owdied, 20 as eae er) 
A second filter is considered which will be noted as the 
least-squares filter which is optimum for uncorrelated 
noise such that the matrix V becomes an identity matrix I, 


and (7) becomes 
Wo= PIPP46@ (11) 


with the corresponding output mean square ¢, for corre- 


lated noise input (case bA). Let 
e2 as ~2 2 
Om Em/ON ) (12) 
bn = On/ON- 


Then the following theorems will be proved: 


Theorem I 


If P,(é) is a polynomial of degree k defined over the 
interval (0, m + a), and N(¢) is a random process whose 
spectral density 1s piecewise continuous and positive, and 
a is the prediction interval of the output of the digital 
filter, then 

42 
Lim Sm = 1. 


mon 


(13) 


Theorem II 


If 6% is the ratio of the mean square output noise to 
mean square input noise for the least squares filter, when 
the input noise is uncorrelated (e.g., V = I, case aA), then 


Lim 3 2470) 


2 
mo On 


where {(0) is the spectral density of the process N(¢t) at 
zero frequency, e.g., such that 


po = 1. 


— [ e'4(n) dr, 


Proof of Theorems 
Let us evaluate the various mean square error ratios 


= WWW 


(15) 
eV nd 
6 = Wwvw (16) 
= 00PP eV Pee ee 
B= Ww a 


l| 


Q’IPP’] PP’|PP')"Q = Q'[PP’)Q. 
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The theorems as stated apply to regression functions 
P,(t) which are polynomials of degree k. The solution 
will be considerably simplified if the sampled polynomials 


are taken as orthogonal with respect to summation.’'*" 
Let 
Pine E.(m =")| 10s deo eee (18) 
Secale ear 
where 
D E(m — ujéj(m — u) = b2:8;, (19) 
u=0 
and 
6:.; — 1 k = ] (20) 
= 0), his 
Then the matrix 
Pies Al (21) 
and (15)-(17) may be written 
bn = Sse aka 
ORV SG (22) 
bn = O10 = DO 
k=0 
The proof of the theorems depends on evaluating 
Lim PVP) = and S| Vea (23) 


mo 


These limits have been evaluated by Grenander and 
Rosenblatt”’ as follows: Since 


RSs E(m + h — wé(m — u) 


Virose 


fT, S =a OL poe 


Pim 2, 


mo 


(24) 
hems Oneal 


Ee —= 0; Une) 


is defined for all integral h, then define a sequence of 
(n + 1) X (n + 1) matrices R, with. elements R{”’. 
A spectral distribution function M(A) of the regression 
vectors may be defined by 


R, 


if et aM). ah) 0) -e seo On) 


8 R. L. Anderson, and E. E. Houseman, ‘‘Tables of orthogonal 
polynomials values extended to N = 104,” Res. Bull., 297 Iowa 
State College of Agriculture and Mechanical Arts, Ames; pp. 297; 
April, 1942. 

9}, E. Allen, “The general form of the orthogonal polynomials 
for simple series with proofs of their simple properties,” Proc. Roy. 
Soc. Edinburgh, vol. 50, pp. 310-320; 1935. 

10 UY, Grenander, and M. Rosenblatt, “Statistical Analysis of 
Stationary Time Series,’’ John Wiley and Sons, Inc., N. Y., ch. 7; 
1957. 
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When h = 0, then 
R, = M(r) — M(—7) = M 


and is nonsingular. Then it is shown that’® 


; = r +7 1 =1 
ae (PVA a= oat bs A amo} ; (26) 
and 
Lim [PVP’] = M72s / f(—») dMQ)M". (27) 
It will be shown that 
aOR A on 
b) dM (A) = 6()Z, (6(A) is the Dirac function) (28) 
c) M =f. 
By (19), 
fee REO Mea i. (29) 


Let &.(m + h — wu) be expanded in a Taylor series where 
the polynomials £,(m — u) are defined continuously over 
the interval [0, (m + h)] 


E,[(m — u) + h] = £,(m — wu) 
h? 


r! 


=e here) at) et niet EO (m =) (30) 


Each term £°?(m — wu) is a polynomial in (m) of degree 
(r — 7), while 


S.=L, T] (m- 9 


j=-r 


is a polynomial in m of degree 27 + 1. Thus the limit 
m— o of each term of the form 


hi ss &(m — we (m — wv 
Vidas: 
= 0(m™'), 


j= 20, 2p (31) 


for finite h. Noting that &,(u) = 0 for u < 0, then Ri” 
is evaluated as follows: 


mm 


Lin, = 
moo u=0 u=m—(h+1) 
En =) 5 (mn = wk | 
of eee en oz 
a ee Gg 
The first term in (82) is expanded into 
- hE Em = WE? Cm — u) 
= he ee 
om a = dl x V8.8; a 


The components of the second term of (33) are seen to be 
zero in the limit, since 


yp fies 


u)é.?(m — u) 


=0 
ASE 


i ies (34) 
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ecause of the orthogonality properties, since &” is a 
olynomial in (m — u) of order r — 7 and can have no 
omnponents of the power (m — u)*, and the summation 
s zero When r > s in the limit as m — © by (31), since 
ae 

The second component of (32) can be shown to approach 
ero in the limit as m “*” because of the finite number of 
erms (4 + 1) in the summation. Thus one obtains 
1 = 6,,, for each finite h, and the corresponding 
egression spectrum consists of only the one point A = 0, 
o that 


dM(v) = d(A)l. (37) 
“hus (26) and (27) are reduced to 
| Lim (PV PY = 2rf(OL 
nd (38) 
Lim [PVP’] = 2xf(0)I 


moo 


D that (22) is given by 


Oe Os, nee 
Lim E = 5 = 2rf(0), eed OR (39) 
mo m m k=0 

nd both theorems are proved. As an example, let N(¢) 
fe an uncorrelated stationary random process, with 
ariance o,. Then in terms of the normalized ratio 6;, 


me may equate the normalized correlation 
f 


m= f fo)a=1, 


nd since f(A) isa constant for all A, (—7 <A < +7), then 


{0 = 5 
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and 


When the linear operator k(r) is defined so that the 
desired output is an estimate of the A/th derivative 
evaluated at m + a, then it is shown in a previous paper’ 
that the exact expression for 6; is given by 


> Ems oT 


L=M Sr 


bees (40) 
Tables and Graphs of 62 are given in the previous paper’ 
for zero, first and second derivatives, for a = 0 (zero lag), 
and for values of m up to 1000. 

A comparison with an asymptotic expression for 6, 
derived by Johnson" is given to enable one to determine 
how large m should be before placing confidence in the 
asymptotic expression. Additional work being done on 
evaluating the mean-square error output for continuous 
filters indicates that the finite memory of the filter m7 
(where 7 is taken as the interval between samples) must 
be large compared to the effective time constant B of the 
correlation function for the asymptotic properties to be 
valid. The effects of nonunity sampling interval (T’ ¥ 1) 
are to replace « by 7/T in (26) and (27) and the final 
solution (389), e.g., 

, bmn 5. 
bien | Sy = 89 


| _ 2nfO), (41) 


f 


mo 


For example, if 
pla) = lV, 


then m7'/B > 1 for (41) to hold. The effects of T ¥ 1 
on 6, for the derivative operator are discussed in a previous 
paper.” 

1K. R. Joanson, “Optimum, linear, discrete filterings of signals 


containing a nonrandom component,” IRE Trans. on INFORMATION 
Turory, vol. IT-2 pp. 49-55; June, 1956. 
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Single Error-Correcting Codes for Asymmetric 


Binary Channels 


WAN H. KIM{ anp CHARLES V. FREIMAN{ 


Summary—In a highly-asymmetric binary channel it may be 
necessary to correct only those errors which result from incorrect 
transmission of one of the two code elements. Minimum weight- 
distance relationships and rules for generating single-error cor- 
recting codes in such situations are given. More code characters 
are generally obtained for a given character length than are obtained 
with codes designed for single-error correction in symmetric 
channels. Examples are given, including one which specifies the 
code which results in the highest average probability of correct 
transmission of equiprobable messages through a highly-asymmet- 
ric channel. 


INTRODUCTION 


shown in Hig. 1 where 


| ET us consider the memoryless binary channel 


A : : : 
a = Pr {any received 0 is delivered as a0}, and 


6 = Pr {any received 1 is delivered as a O}. 


DELIVERED SIGNAL 
(OUTPUT) 
O 


RECEIVED SIGNAL 
(INPUT) ol 
O 


(1-A) 


G25 


Fig. 1—Memoryless binary channel; 0 < a, B < 1.0. 


Silverman’ has shown that a channel characterized by a 
and 8 may be used to transmit information if a # 8, but 
that channel capacity is very low unless a > 68 or B > a. 
(For example, if a = 0.6 and 8 = 0.38, the channel capacity 
is 7 per cent of that of a noise-free channel transmitting 
the same average number of symbols per second.) For 
the present we will assume only that the relationship 
between output (delivered) signals and code elements is 
such as to make a > 8. 

If in transmitting a code character, k of the received 


* Manuscript received by the PGIT, September 22, 1958. This 
work was supported by National Science Foundation Grant G 3676. 

+ Department of Electrical Engineering, Columbia University, 
New York, N. Y. 

1R. A. Silverman, “On binary channels and their cascade,”’ 
IRE Trans. ON INFORMATION THEORY, vol. IT-1, pp. 19-27; 
December, 1955. 


ones are delivered as zeros and / of the received zeros are 
delivered as ones, we will say that both a k-tuple 1-error 
and an I-twple 0-error have occurred. When the channel 
has symmetric transmission properties, the general 
requirement that a # B may be reduced to a > 0.5. As no 
distinction is made between 0-errors and 1l-errors in a 
symmetric channel, it follows that all k-tuple errors are 
equally probable and (& + 1)-tuple errors are less likely 
than k-tuple errors. 

In asymmetric channels, however, (k + 1)-tuple 1-errors 
may be more probable than k-tuple 0-errors. For example, 
110 will more likely be received as 000 than as 111, 
provided 

2 


B 
el i) 


In what follows, it will be assumed that the channel is 
highly-asymmetric with B > (1 — a). 


ap’ > (1 — a)(1 — 6)’, or (1 —a) < (1) 


ERROR-DETECTING AND -CORRECTING CoDES 


In the coding problem we shall discuss, character length 
and error tolerances are specified and we seek to maximize 
the number of different messages which may be sent 
subject to these conditions. This is essentially the problem 
considered by Hamming” and Golay* among others, for a 
symmetric channel. In the symmetric channel case, error 
tolerances result in requirements such as “correct all 
single errors and detect all double errors.” For highly- 
asymmetric channels, however, the resultant requirements 
will be of forms such as ‘‘correct all single l-errors and 
detect all double 1-errors.” 

Hamming established the minimum distance require- 
ments between input code characters for error-detecting 
and-correcting codes. These are sufficient to insure k-tuple 
error correctability or detectability in any type of channel 
but may be relaxed somewhat when only k-tuple 1-errors 
are to be considered. The nature of this relaxation is easily 
seen in the case where a and 6 are such that we may neglect 
Q-errors and correct only single l-errors. Pairs of code 
characters must be at least three-distant when all single 
errors are to be corrected but, in the case of single l-error 


> Operating experience has shown that certain magnetic tape- 
storage units exhibit just such transmission properties. 

*R. W. Hamming, “Error-detecting and error-correcting codes,” 
Bell System Tech. J., vol. 29, pp. 147-160; April, 1950. 

4M. EK. Golay, “Binary coding,’ IRE Trans. on InrorMa— 
TION TueEory, vol. I'T-4, pp. 23-28; September, 1954. 


959 


orrection, pairs of code characters need only be two- 
istant provided they are of the form 0OX XX and LIX XX. 
hus when 1OXXX or 01X XX is delivered in the asym- 
ietric case, it is assumed that the input code character 
vas LIXXYX: 

If W(x) is the weight (number of 1’s) of code character 
fF and D(x, y) is the Hamming distance between code 
haracters x and y, then we may use the fact that 


D(a, y) = | Wi) — WY) | (2) 


© establish the requirements for single |-error correct- 
jpiaty in the form 


D(x, y) 
| W(x) = Wy) | 


IV 


Ss Ore a (3) 
(4) 


br all « and y. Any weaker requirement such as (4) may 
ilow an increase in the number of different sequences 
hich can be sent with a particular character length. 
i will describe a coding scheme below which makes use 
sf (4) to achieve such an increase. 

When 6 is much greater than (1 — a), error tolerances 
nay require us to correct k-tuple l-errors while neglecting 
ll 0-errors. The minimum weight-distance requirements 
or pairs of code characters of a k-tuple l-error correcting 
‘ode are 


IV 
iS 


| Wa) — Wy) | >k+1, or (5) 
DG eee aa Ne WV (ay) = Wy), (6) 
or alle and y where |W(x) — WYy)| = 0,1, ---, le 


i 


SINGLE 1-ERRoR CORRECTION 


Let us assume for the moment that the characters of 
e single l-error correcting code are to be of even length, 
m. We first specify code character prefixes of length m 
»y forming all possible m-length binary sequences. lor 
xample, if 2m = 6, the prefixes would range from 000 to 
11. Suffixes are generated for a given prefix by adding 
hat prefix to code characters of an m length single error- 
correcting code. The addition is performed position by 
osition, modulo two, and the m-length code is taken to 
e a single-error-correcting Hamming code formed with 
ven parity checks. Thus, when m = 3, 000 and 111 are 
e single-error-correcting code characters and 110, as 
. typical prefix, will be combined with suffixes 110 and 
01 to give code characters 110110 and 110001. 

In order to detect and correct errors at the output of 
e channel, we add the prefix of the delivered signal to 
e suffix and test to see if the sum is one of the code 


6 Further discussion of multiple l-error correction and detection 
ay be found in “Multi-Error-Correcting Codes for Asymmetric 
‘hannels,”’ to be given at the International Symposium on Circuit 
i. Information Theory, June, 1959. 
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characters used to generate suffixes. For example, if 
110100 is delivered by the channel, we take 


110 
@ 100 (7) 
010 


and deduce that an error has occurred in the second or 
fifth position. As the channel is such that 0-errors may 
be neglected, we assume that a l-error has occurred and 
that the input signal was 110110. Note that the symbol- 
correcting nature of the correction scheme depends upon 
the use of an m-length symbol correcting code in generating 
suffixes. It is for this reason that the Hamming Code was 
chosen even though it is sometimes possible to generate 
more suffixes for a given prefix through the use of single- 
error message-correcting codes.” 

The rules of generation are explicitly stated below and 
are followed by two examples of code generation. The proof 
that all pairs of code characters so generated satisfy (3) 
or (4) is included as the Appendix. 


Let us use the following notation: 


n : code character length: n > 1. 
m  : prefix length: 
m = n/2 when n is even. 
m = (n — 1)/2 when n is odd. 
Suffix is, therefore, of length (n — m). 

: set of all single-error-correcting Hamming code 
characters of length (n — m) formed with even 
parity checks. H, is that element of [H] consisting 
of (n — m) 0’s. When (n — m) = 1 or 2, [H] = Hp. 

h,-»: number of elements in [H]. 

@_ : position-by-position addition, modulo two. 


[| 


The rules of generation are then 
1) Form all binary characters of length m. These are 


the prefixes. 


2) Supply suffixes for each prefix of even weight, Pe,;, by 
forming (Pe; ® H;) for each element of [H]. When n 
is odd, append a O to the right end of Pe, before 
adding. 


3) Supply one suffix for each prefix of odd weight, 
Po:, by forming (Po; © Ao). 


Example A:n = 8,m = 4, hn-n = 2 


I 


[H]: H, = 0000 


H, = 0111° 


6 Hamming’s double-error-detecting code may be used at this 
point. H; would then be 1111 or four-distant from Ho. 
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TABLE I (EXAMPLE A) 
Prefixes of Even Weight Prefixes of Odd Weight a 
“0000 0011 0101 0110 1001 1010 1100 1111 | 0001 0010 0100 O111 1000 1011 1101 1110 
| @ Ho | 0000 O11 0101 0110 1001 1010 1100 1111 | 0001 0010 0100 O111 1000 1011 1101 1110 
ami - @H, | Oli1 0100 0010 0001 1110 1101 1011 1000 


Thus the code words obtained with H, are 


Example C:n = 9 Example D:n = 9 


00000000 10011001 00010001 10001000 1) 000100110 is delivered. 1) 000100001 is delivered. 
00110011 10101010 00100010 10111011 
01010101 11001100 01000100 11011101 2) 0001 2 me OOO L 
01100110 ALITA 01110111 11101110 00110 00001 
and those obtained with H, are 00100 00011 
00000111 10011110 
00110100 10101101 3) Hamming check indi- 3) Hamming check indi- 
01010010 11001011 cates error is in 3rd or cates error is in 3rd or 
01100001 11111000. (m + 3)rd position. (m + 3)rd position. 
EEN ie BI tah: 4) No sum modulo two 4) Sum modulo two of | 
[H|: H, = 00000 required. prefix is odd. 
Hy 200001 
He" 100i 5) Correct character is 5) Correct character is 
fie VPA) 001100110. 001100001. 
TABLE II (EXAMPLE B) 
Prefixes of Even Weight 
0000 0011 0101 0110 1001 1010 1100 Tait 
® Ho 00000 00110 01010 01100 10010 10100 11000 11110 
Suffixes @® Hi OO111 00001 01101 O1O1LL 10101 10011 L111 11001 
@ H, 11001 GEL 10011 10101 01011 01101 00001 OO1LL 
@ H; 11110 11000 10100 10010 01100 01010 00110 00000 4 
Preffixes of Odd Weight 
0001 0010 0100 O111 1000 1011 1101 1110 
Suffixes @® Ho 00010 00100 01000 01110 10000 10110 11010 11100 


Ru.LEs FoR ERROR CORRECTION 


Align the first position of the received code character 
with the (m + 1)st position. Perform a position-by- 
position addition modulo two and apply a single-error- 
correcting Hamming check on the sum. If one position is 
incorrect in the sum and it corresponds to the addition of 
a zero to a one or a one to a zero, replace the zero by a one. 
If the incorrect position corresponds to the addition of a 
zero to a zero, sum the prefix modulo two replacing the 
zero in the prefix if this sum is odd and replacing the zero 
in the suffix if this sum is even. 


NuMBER OF CopE Worps OBTAINED 


The above system yields (h,_,, + 1)2” * code characters 
of length n where h,_,, 1s the number of code characters in 
a single-error-correcting Hamming code of length (n — m). 
When n does not correspond to a close-packed’ Hamming 
case, (h,-m + 1)2””" is equal to (h, + 2” *) where h, is 


7In this context, close-packed implies n is of the form 27 — 1, 
a = 0, 1, 2, +: . Further discussion of close-packed codes may be 
found in C. Y. Lee, “Some properties of nonbinary error-correcting 
codes,’ IRE Trans. on INForMatTIoN TuEory, vol. IT-4, pp. 77- 
82; June, 1958. 


e number of code words in a single-error correcting- 
amming code of length n. Less than h, code characters® 
e obtaimed in close-packed Haniming cases, but slight 
odification of the above method enables h, code charac- 
rs to be generated. Table III lists the number of code 
aracters obtaimed with various coding schemes for 
lues of n between two and sixteen. 


TABLE III 
Number of Code Characters in Single-Error-Correcting Codes 
Hamming Code | Single 1-Error Correcting Code | Golay Code 
2 2 
2 2 
2 Ph) 
y 2 EL 
23 23 4 22 
24 2% + 92 
2! 24 +4 93 20 
2 Q +4 23 38 
28 26 + 24 68 
27 PAL + oA 
28 98 + 25 
99 Q9 4 95 
210 210 + PAL 
ou 210 + O86 
91 Dulien yn 
| Note that when n = 3, the use of [H] e Ho enables hn code characters to be 


merated even though this is a close-packed case. In other close-packed cases, 
e Hamming Code itself should be used. 


Correct TRANSMISSION OF EQUIPROBABLE MESSAGES 


The codes described thus far have sought to maximize 
he number of possible messages for a given character 
mgth and error tolerance. We shall now briefly consider 
ne case where character length and number of messages 
e specified and it is desired to maximize the probability 
* correct transmission of these messages. Slepian’ has 
psigned codes for use in a symmetric channel which are 
otimal in the latter sense for many combinations of 
naracter length and number of messages. In example E 
e present one of the Slepian codes which is optimal for 
ur equiprobable messages and character length four. 
message is now said to be transmitted correctly if any 
' the output signals assigned to it is delivered by the 
nannel. Note that every possible output signal has been 
ssigned to a message even if no one message is most likely 
» have resulted in a particular output signal. Thus in 

ample E, 1010 is equally likely to have been caused by 
lessage B or message D but has been assigned to message 


J 
», 


1) Symmetric channel, 8 = (1 — a). 
2) Four equiprobable messages (A, B, C and D). 
8) Code characters to be of length four. 


8 The actual number is (4h, + 2”) which can be shown to be 
33 than An. 

ED: Slepian, ‘“‘A class of binary signaling alphabets,” Bell System 
ech. J., vol. 35, pp. 203-234; January, 1956. 
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One of the Slepian codes which results in highest prob- 
ability of correct transmission under these conditions is 
found as Table IV. No rules are known which will gener- 
ally yield such optimal codes when 8 > (1 — a) but, in 
the particular case of four equiprobable messages and 
character length four, it can be shown by exhaustion that 
the code described in Example F is optimal. 


TABLE IV 


Message A B C D 


Input signal 


assigned to the 0000 1011 0101 1110 
above message. 
CODE § _ —————-— 
Output signals 0000 1011 0101 1110 
assigned to the 0100 1111 0001 1010 
above message. 0010 1001 O111 1100 
1000 0011 1101 0110 


Example F: Given: 
1) Asymmetric channel, 8 > (ee — a). 
2) Four equiprobable messages (A, B, C and D). 
3) Code characters to be of length four. 
The code which results in highest average probability of 


correct transmission under these conditions is found as 
Table V. 


TABLE V 
Message A B C D 
Input signal 
assigned to the 0000 0101 1010 1111 
above message. 

CODE A Output signals 0000 0101 1010 1111 
assigned to the 0001 0010 1110 
above message 0100 1000 1101 

1011 
O111 
0011 
0110 
1001 
1100 


Note that the output code characters above the dashed line correspond to those 
generated by the rules for a single l-error correcting code as given above. The inclu- 
sion of the characters below the line naturally requires that the correction scheme 
be considerably more complex than before. 


Although no general conclusions may be drawn from so 
specific an example, it is still of interest to investigate 
under what conditions Code $ and Code A are of equal 
value. In Table VI we have assigned values of 8 to the 
symmetric channel and have then calculated values of 
(1 — a) and @ for the asymmetric channel such that the 
probability of correctly transmitting any of the four 


messages through the latter using Code A is slightly higher 


than the probability of correctly transmitting that message 
through the symmetric channel using Code 8. 
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TABLE VI 


Symmetric 
Channel 


Asymmetric 
Channel 


Sey 6), B Csr) 
1.0 X 10°6 700 X 10° =| Ss 0.25 & 10-8 
[Oe LOm? 20 X 10-3 | 0.25 X 1073 
1.0 X 107 70102 | 0.25 X 107 
04 | 0.25 | 0.032 


| 


Note that 0.25 is sufficiently greater than 0.032 to insure that the least probable 
double 1l-error in Code A is more likely than the most probable single O-error. 


CONCLUSION 


The techniques of establishing minimum weight-distance 
relationships through use of shorter error-correcting-code 
characters has been used to generate multiple 1-error 
correcting codes as indicated above. Karly work indicates 
that the problem of designing symmetrical multiple-error- 
correcting codes may also be profitably approached in 
this way. 

The choice of a code for a particular system usually 
involves the consideration of many factors and it is 
therefore generally not possible to determine under what 
conditions an asymmetric code is desirable. It would seem 
that asymmetric coding should be considered whenever 
prototypes exhibit highly-asymmetric transmission charac- 
teristics or whenever such characteristics depend upon 
the setting of a bias or threshold level. 


APPENDIX 


In order to show that the rules given above result in a 
single l-error correcting code, we shall use the following 
relationship: 


June 


D(x, y) =a | 

Dix OBz,y@w) = b 

Ge OMI ee “| 

[ De, w) = |a—b| + 2c 
c= 0,1, <2 MUNGO. 


where n is the length of binary characters x, y, 2 and w. 
The proof of (8) follows directly from the fact that the 
minimum value of D(z, w) is |a — 6| while the maximum 
value is the smaller of (a + b) and n. In the special case 
where a = 0, we have 

Dix O®z,x Pw) = DE, wv). (9) 
We use (9) to show that two-code characters generated 
under the above rules having the same prefix satisfy (3) as 
all elements of [H] are at least 3-distant. 

If two code characters have different prefixes but both 
prefixes are of even weight or both are of odd weight, the 
prefixes must be at least 2-distant. Using (8) we can show 
that the suffixes of 2-distant prefixes are at least 1-distant 
and hence the code characters satisfy (3). 

Finally we consider the case of one code character with 
prefix of even weight and one code character with prefix 
of odd weight. When these prefixes are |-distant, (8) may 
be used to show that the suffixes are at least 3-distant 
unless they were both formed by the addition of Hy. But 
when H, has been added to both prefixes, the resultant 
code characters differ in weight by at least 2 and (4) is, 
therefore, satisfied. 


(8) 


implies 
—a,n — Dd) 
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Summary—The present study concerns the optimal filtering of 
class of input time series in which the amplitude is modulated by 
niformly-pulsed periodic functions. A uniform sampling of the 
Lt at a period equal to the pulsing period displays the property 
_ time invariance. The consequent usage of bilateral Fourier- 
Aplace transformations and the separability of terms implicit in 
(e time-invariant nature of the processing effectively inverts the 
fiener-Hopf equation and solves the problem. The weighting 
nection is shown to be the sum of the Wiener-Zadeh-Ragazzini 
lution for the unmodulated case and set of appropriately-weighted 
lta-function-derivative terms occurring at each end point of the 
hising intervals. 


TTNHE title could equally well be “Optimal Pulsed- 
Memory Filtermg of Stationary Time Series,” 
since the form of the optimizing condition is that 
br a pulsed-memory filter. The difference lies in the 
aterpretation; the pulsed-memory filter characteristic is 
ro outside the pulsing intervals, whereas in the pulse- 
nodulated input case the weighting function is generally 
ionzero for all positive time. 

The comparative length of the present study has 
esulted in its presentation in three parts. 


| 


Part I—Derivation of Pulsed-Memory Wiener-Hopf 
ntegral Equation. This part contains the Introduction, 
formulation of the Problem, and the initial step in the 
iaalysis where the optimizing condition, in the form of the 
Viener-Hopt equation, 1s shown to involve a_pulsed- 
nemory filter. 


Part Il—Derivation of Optimal Pulsed-Memory Filter. 
‘his part contains the complete analysis and inversion of 
he optimizing integral equation. The solution is shown to 
ake the form of the conventional Wiener solution for the 
nmodulated, unpulsed solution, plus a sum of appro- 
riately-weighted singularities, delta-function-derivative 
erms to compensate for the loss in area in the convolution 
rtegrand due to pulsing. A Review of Solution Procedure 
: included to summarize the method. 


| Part I1I—Finite Memory Considerations. This part 
iscusses finite memory considerations still applicable to 
ulsed-memory filters. An example of the infinite memory 
tationary input is included. 


- * Manuscript received by the PGIT, December 8, 1958. Title 
nd modification of Part A of Doctoral Dissertation, University of 
‘alifornia at Berkeley, Department of Physics, September, 1958. 

+ Convair (Astronautics) Div. of Gen. Dynamics Corp., San 
diego, Calif. 
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Optimal Filtering of Periodic Pulse-Modulated 


Time Series” 


A. JANOSt 


Part I—DerrtvaTion or PutsEep-MEmMorRY 
Wiener-Horr InrecraLt EQuarion 


Introduction 


The following is an investigation of the general problem 
of obtaming a linear time-invariant system to perform a 
mean-squares “‘best’? approximation to a desired linear 
operation on a periodic pulse-modulated time series. This 
time series will consist of a stationary, ergodic random 
part and, at most, a sure function made up of an arbitrary 
linear combination of time translation-invariant elements. 

The theory used has been developed in its applicable 
form by Norbert Wiener [1], and extended by various 
others [2, 3]. Whereas previous work has been concerned 
with samples of the past of the input function over a 
continuum of time points and a discrete set of time points 
as separate problems, the present case has to do with a 
special combination of both types of sampling. It will be 
shown, however, that the problem is reducible to that of 
solving a generalized form of Wiener-Hopf integral 
equation. The solution technique for this equation will 
then depend in part upon the methods of Wiener and the 
others. 

With the class of input functions /; which undergo a 
periodic modulation, there is associated the class gf; 
where g is the periodic modulating function of period 7’, 
4; 


ga) = g(a te mel) 


Consider the response of a linear time-invariant system, 
of impulse response W(r), to this periodically modulated 
input gf’;. 


PO = [We glP ie) ar 


or 


F(t) = i Woot — pm eee 


Now let the input functions be shifted in time origin by mT, 
FG) are tame) 
then 


Fut) = le WG ge HG non Peds 


timT 


/ (oP Alene AMEN Ayah: 
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N 
Le fie(t) = >> ay f,(t), where N and the f, are known. 


Unmodulated input 


T+h 2T 


ltt -T+h bh v. Aeon. hu oF 


Periodic modulator 


£,(Ms@M 


2T+h 3p T 


Modulated input 


Fig. 1—Input and output waveforms of the periodic modulator. 


But since g(r) has a period 7, 


Fit) = i BG ner Nec ee Fie, 


or 
Fis + mT) D F(t + mT). 


Therefore, the system is time-invariant with respect to 
input-time origin translations of m7’. In order to obtain 
this condition in a physical system, it is necessary to 
sample the output uniformly with a period 7’. 

Since the process of periodic output sampling reveals a 
time-invariant character, the subsequent investigation 
will deal only with this sampled case. It is expected that, 
for most cases, this possible oversimplification will be 
compensated for by the greater generality of the results. 
Hence the over-all operator is to filter a piecewise con- 
tinuous time process into a discrete time-ordered sequence. 


Formulation of Problem 


The formulation of the optimal filter problem is the 
customary one [1-3]. Given an input function, 


fi = BO =F Re ah VEN 


the true signal consists of a stationary random part, 
fim, and a nonrandom part, 


1 


The noise, fin, is assumed stationary and random. It is 
further assumed that all second moments of the joint 
processes (f;,, fin) are known, 2.e., 


(fio Ofiolt ae T))5 


Introduction of the condition of zero means for f;,, and 
fin amounts to a subtraction of their steady-state rms 
values. With this added condition, the problem is reduced 
to that of obtaining an optimal filter to operate on the 
functions f;, and the unpredictable parts of the imput. 

The optimum criterion is in the sense of minimum 
variance of output minus desired output. 

The output is given by 


for all r and P, 0 = m, n. 


jot) = fade We DoHA®). 


W(r) is the impulse response (generalized one-sided 
retarded Green’s function) of the linear time-invariant 
system, and g(r) = g(r + mT), the periodic input modu- 
lator. 

Given a desired operation on the signal part of the input, 


foot) = [ D) inl — 7) + fat — 9} dr. 
The error is to be minimized in the mean-square sense, 


ai) = fold) — fold 
[awe - nortan 


_ i dr Liz) {fst — 7) + finlt — 7} 
On taking the ensemble mean, 
e) = [ar We= Dahl) — fdr LU = F.0), 


since the means of both f,, and f, are by construction 
ZeYO, 2.€., 


(fn(Q)) = (f()) = 0. 


Now if we set ¢ = mT’, T being the period of g, and then 
impose the condition of zero mean on e, 


Conte / i dz Wl = DaGie 


= io dr L(mT — 7)f.(2) = 0, 


or 


[ Pec O 


= [ a iOn Gr a 


.(7) is a linear combination of time translation-invariant 
ements f,(7), 7.e., f:.(7) may be considered a vector in a 
ute dimensional function space 3, which remains un- 
anged with respect to translations in time, 3, = 3,4,7. 
xamples of such a space are the set of all polynomials 
a known degree, or more generally, the set of eigen- 
pnctions of a constant parametered differential opera- 
or [3]. 

Let a choice of the base vectors for a given 3, be 
1) fe, iy Ag fy) then 


N 


f:mT = T) = SS anf(mT ae. 7) 


1 


t since J, is invariant 

| N 

f.(mT — 7) = 2D S (—af. (mT), where S (—7) 
L kk! kk! 


| the k, k’th matrix element of the representation of the 
translation operator in 3,. Thus 
N N 
fmt — 1) = >) dia, S (—afr(ml) 
kk’ 


k=1 k’=1 


d (1) becomes 


D aafe(mt) [dr WO)g(=2) 8 (=9 


SnD) 1. dr Lr) 8 (—2) 


jomion kk 


thich holds in general for 


dr WO)g—7) 8 (=7) 


t 


(2) 


r 
-| Pade pane 
| —o kk’ 


set of N” equations of constraint on W(z). 

Now we must consider the pulsed nature of g(r). 
7) = og + mT) and g(r) = Ofor mT > +r = mT +h, 
ll m integers, or g(—r) = 0, mT <7 < mT — h. Thus 
is the duration of each pulse and g(7) is represented as 
1 infinite train of such pulses. 

If u(r) symbolizes a train of unit pulses, 


Cpe le nl <r int hy allen 


= 0, otherwise. 


1s g(r) = g'(r)u(r) and g’(r) can be considered equal 
g(r) over the nonzero pulse intervals and continuous 
1 between. Hence (2) may have the form 


W)g'(—Au(=7) 8 (2) dr = ip La) 8 (=1) dr 


nd on setting 


W(t) = W(2)9'(—7) (3) 
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the equations of constraint become 
fig dr W(x)u(—7) 8 (=7) = ie dr U(r) 8 (1) dr 
he eee Ne (4)" 


On recalling the expression for the error and using the 
periodicity of g and definition (3), 


Ney — [ de WG aa Tae 


— [ar LOOT — 2) + fin? — 9} 
and with (4) 


er?) = le dr W(r)ul— 7) f fim(rT —7r)+f..°7T — 7)} 


= i : PIREN i >) 


So the error is a linear functional only of the stationary 
random processes f;,, and f;,. Thus 


rl) = hk ; ‘| _drdc Waianae 
‘{feml?T — 1) + fin — 1} 
Ate EL — 72) ett een 
259 i % [ ; dn da! WG\L Gee 
“fen — 1) + fT — )VfinT — 1/) 
ae { [ar Dota - of 
and on taking the ensemble expectation, 


(rT) = [ ; / “dreds WO) Wal ou ae eee 


—2 i {he dr dr’ W(1)L(1!)ul— 1) X(r —7r)+kp 


where ¢(r) 1s the autocorrelation function of f;,.(7) + fin(7), 
X(r) the cross correlation between f;,.(7) + fi,(7) and 
fim(7’), and kp is the mean-squared value of the desired 
random expression 


i : ue V ee =), 


which is positive, independent of W(r) and constant. 
Due to the property of time invariance, the mean-squared 
error is independent of time and the output is stationary. 
Thus the expression to be minimized with respect to 
W(r) under the constraint (4) becomes 


1 Tn order for the integral on the left to converge it is sufficient 
that S,,,(—7) and (7) be square integrable over (0, ~). 
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[ i. dr dr W(r)W(7’)u(— r)u(—7')o(7 — 7’) 


vi) v0 


—2 iP [ dr dr’W(n)L(r)u(— 1) X(7’ — 7) 


if drW(r)u(—71) S (—7) 


J 


_ Ir drL(r) S a}. 


W—W.:. + dW, then eu = 0 
yields the minimal condition, 


[ar a)u(=nu(= oC" — 7) 


co 


= / drL(7)u(— 7’) X(7’ — 7) 
+ YS rv\wul—7r) S (—7) for 7” > 0 (5) 


where the ),,, are as yet N* undetermined Lagrangian 
multipliers. The above, characteristically, is a modified 
Wiener-Hopf integral equation. 


Analysis 


Let us set 


Wr) =f ab)XC — 2) + Dw 8 (-79); © 


then (5) becomes 


w= drW (nul —7)o(7’ — 1) ue) = (0), 


roe AOE (7) 
Notice that W(r)u(—7r) is zero for all r < 0 [since 


W(r) is physically realizable] and for mT <r < mT — h, 
for all integers, m. Thus if we set 


W(2)u(— 7) = W(r), W(r) (8) 


is a pulsed-memory filter. The solution of (5) is contained 
in the solution of 


eat dtW(1)o(7’ — 7) ue) — (0), 


(J 0 


TOT tae) (9) 


where W(r) = Oforr < Oand mT < + < mT + fh, all 
m integers. 


Part [II—DerrivaTion or OprmMaL 
Puusep-Memory FIuTer 


It has been shown that the optimizing condition for a 
uniformly-sampled output with period equal to the pulsing 
period T is of the form of a modified Wiener-Hopf equation 
involving a pulsed-memory filter, namely, 


Jun 


“rf drW(r)¢(7’ — 7) — ua} Set (ae a eee) (9 
JQ 

where W(r) = Oforr <OandmT <7 < mT +h, allm 
integers. 

Now for most cases ¢(7) may be considered a linea: 
combination of exponentials, e “"'"! so that the bilatera 
Laplace transform (generalized Fourier transform but 
for an argument phase difference of +/2) becomes 


ir dro(ne”” = &(p) = ¢(p)e(—p) = en (p)en( =p) 


Y OMe 


where ¢y(p) and gp(p) are polynomials in p, the degree 
of ¢p(p) exceeding that of gy(p) at least by unity; further 
both ¢gp(p) and ¢gy(p) have zeros only in the right-half 
complex p plane. 

If (zr) were completely arbitrary for 7 > 0, instead of 
piecewise zero, the solution of (9) obtained by well 
known techniques [1, 2] would be 


Lp ie \ 
a7 dpe” AY = W 1 
Qri Jia” op) \o(—p)) « 7) 
where V(p) is the transform of (7) and the plus subscript 
indicates analytic continuation in the right-half p plane 
of the expression within the brackets. Let us conside1 
the transform of (9) with u(—r) absent. Then, ] 


Y(p)®(p) — V(p) = G(p) 


where G(p) is bounded and analytic in the left-half p plane. 
Or, on referring to (10) and transposing, 


Y(p) 
en(P) 


"W(r) DP = 


(12) 


p( —P) (p) = p(—P) 
gy(—p) gn(—p) 


gy(p) — Gp) = Gp). 8} 
Notice that (9) and (12) imply that G(p) has the poles 
of 1/gp(—p) so that G’(p) can only have the poles of 
1/gy(—p) plus, at most, those of a polynomial in (—p) 
which can be regularized by setting 


(—p) = lim (" _ “). 


€>0 € 


This corresponds to the limit of a finite difference in the 
time domain. Thus G7/,, is also interpreted as analytic 
and bounded in the left-half plane. 

Y(p), the transform of W(r) is zero for negative argu. 
ment, 7 < 0, and piecewise zero for positive argument 
t > 0. However, Y(p)/¢p,,) 1s the transform of a smootk 
time function for positive time, 7 > 0, being zero fo 
negative argument only. Thus, on taking the analytic 
continuation in the right-half p plane of (13), 


Vip) quan {pew ad 

gop) ex(p) | gx(—p) J. 
The inverse of (14) is defined by 

ci ee a : 

Dnt Bes vp(p) = Wi(7). (15 


ow the actual solution W(r) is obtained from W,(r) 
, the operator ¢p(p), 7.e., W is the “driving function” in 


gow, ( T) 


t over any interval of definition of (16), possibility 
initial conditions must be allowed. Thus, over each 
Sen! —h ir < mT’, WW is nonzero and a set of initial 
ynditions may be introduced at ¢ = mT — h. Over 0 < 
T <7 << (m+ 1)T — AW is zero, (16) ishomogeneous, 
nplying another set of initial conditions at 7 = mT. 
hese initial conditions, manifested as 6-function deriva- 
ives at the pulse end points, must be allowable because 
' the arbitrariness of the W,’s solving (15). Another 
jay of expressing the same thing is the following: 

It is evident that W(r) depends only on that part of 
iS tora and ml —h <= 7 mlm > 1p sinee 
?(r) may be expressed as u(—7r)W,.( ), where u(—r) 
-a train of unit pulses which are zero over mT’ < 7 < 
‘Tl’ — h. Thus, symbolically, from (13), 


= W(r). (16) 


W(7) = en(p) (u(— 7)W(7)} (17) 


there the Heaviside operational notation is used to 
ppresent a differential operator. Now the action of the 
erator on the expression within the brackets is of two 
farts, one referring to the nonzero, piecewise continuous 
‘ehavior between the discontinuities at m7’ — h and 
»T, and the other related to the discontinuity points 


pee lvcs: On using (15) with (17), it follows that 
L 7) iD nen ev(P) {else Ho) 18 
Cys 201 sha De en(p) gn(p) i 


L s 3 {Gnron.c8 (r — (mT — h)) + an7,,6 (7 — mT)} 


m=1 


yhere the a,,7-,,, and a,,7,, are constants, the r denoting 
e rth derivative of the corresponding 6-function. The 
function terms arise from differentiation of u(—r)W, (7) 

t the discontinuities and D is the degree of gp(p). The 
onstants are to be determined to satisfy the integral 
quation (9) when (16) is substituted [2]. 

Consider the first term on the left side of (18); by 
eferring to (10) and (11) we see that formally this is 
he optimal Wiener filter for a continuous input, then 


Ae “ice 2 ee See Oe ame) 
nd the integral equation (9) takes the form 
i u(—7)W(7)b(7’ — 7) dr 
+> payee son) 
rec) 
= v0") =f Wola)olo" — 2) dr 
GornT —h <7 < nT, allw> 1) (20) 
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from (19). But, 


ibe Wo(r)b(7’ — 7) dr — iP u(—7)Wo(1)b(7’ — 7) dr 


= i iGWEG ae (21) 


where v(7) is the complement of u(—7r); thus 
1 = w—7) =1G) = 1, ml <r<mT—A 


otherwise. 


Hence with the above expression, (20) becomes 


ss SS (Sor - 9) 


m=1 r=0 


T=mT—h 


+ Omr,-(—)’ (4 CG = >) 


= [ awonee! = 2) ar, 


Cl oe ama 


Thus let 7 = qf + 7 —h,0 < 
takes a more eS form 


pm ofa Amt —n rk (5 pp CLG ae 


ally aie a0" (22) 
7 =< hq = Vrands@2) 


GE =) 
+ aur. (-Y(S oar + # = 7-0) } 


~ Wolm? + 7 oq =) am) 


m= 2d 0 


bg! = gl aT Widila Oana 


The right side of (23) may be given interpretation as 
a sum of decay terms for a fictitious system with various 
delays and of impulse response ¢(7); the “driving function” 
W (7) is essentially switched off at the times of observa- 
tion, gf + 7’ — h, 0 < 7’ < h, and only the poles of 
(7) are- really determining the continuous behavior of 
the output response. The set of coefficients and derivatives 
of the left side are also, effectively, decay terms depending 
on $(r). 

Now, as mentioned previously 
a series of exponentials, 


[see (10)], d(7) is assumed 


= Dy Cxbx(7), b(t) =e, (24) 
Thus, 
bx(t) = x(t) + x(7) (25) 


where the two components $x and ¢x are zero for negative 
and positive argument, respectively. It also follows that 


bx(t + 7’) = bx(r)bx(7’); both (26) 


Further, W(t), the solution for the non-pulse-modulated 
case, may generally be expressed as a sum, 


W (7) a %, Wx(7), 


tS 0s 
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for we(r) such that 


Wo(t a 7’) — >y Wl 7’ )Wx(7) 5 both ae tf > 0; (27) 
K 
that is, Wo (7) is a vector in 3,. The sequences a,,7-,,, and 
Qnr,r, by virtue of their reference to initial and terminal 
points of each pulsing interval of u(—7) for positive 7, 
[from (18) since ‘W is realizable], necessarily must be 
physically-realizable digital-filtering processes, 7.e., 
OnT—h,r 7 0, m << 1 


m < 0. (28) 


Substitution of (24) through (28) into (23) and con- 
sideration of the positiveness of 7 — h and 7’ [in reference 
to (26) and (27)] obtains 


Es 2» os (On Poin TOG te OK(L = 1) }( =O) 


AmT,r = 0, 


= Even — 90) [ we(oeee ar} 
“ox ((q — max | 


ai | Se ye {> NO ra ee Ac Oars hese aa h) ag 


=q+l1 K ie 


= a EG yr) | © Belabx(7) ar} 


“oK((q — meat) | oil 


all g> 1. (29) 


In order for the above to hold for all g > 1, for a time- 
invariant process, terms corresponding to the +, — 
superscripts must be equated to zero separately. Thus 
each of the square-bracketed terms must equal zero. For 
such generality, it is then further necessary that all terms 
with a given m be equated to zero. This produces a pair 
of equations for each m. 


Da os {Qm7—h,r ie Oim—1) 7, PKL 7 h)}(Fax)’ 


— ¥ we((m — yr) [ BE DETO) ir} 


‘b«(7)ox((q — mT’) = 0. 


forall gem.) (30) 


[the pair of equations implied by the superscripts (+)]. 
Now, since the limits of the 7 subscript terms are from 0 
to D and there are also D + 1 of the K’s, this pair of 
equations, (380), may be separated into a system of 
D + 1 pairs, one pair for each K. Thus we have the simpli- 
fied system, which for each given m is independent of gq. 


ss { Omrenge Comair Oe — h)}\( an) 


r 


=D wm ((m = 12) i Be (a)bR(2) de 


June 


DE {Qniti—h,> ae Hao ml = h)\(ax)' 


= DS wx ((m — 1)T) i ni Wx (t)px(7) dr. 


Ko =.0, 1, eee ee ce (31) 


The author has previously obtained the above system 
of equations by recourse to the z transform [4] over the 
time points m7’ for the various sequences concerned in 
(29). However, it is felt that the above discussion (actually 
of the implications of time invariance), is as sufficient 
and more condensed. 

The coefficient matrix for the a,,7_,,, and @,,7,,’8 1s 
generally nonsingular, which is to be expected for distinct 
Bip 

Thus, the evaluation of the 6-function derivative term 
coefficients is reduced to solving of a simplified, consistent 
set of linear equations (31). With such a solution set, 
(9) is formally solved but ¥(r) in the nonstationary case 
is a function of N’ Lagrangian multipliers \x«- [from (6)]. 
A substitution of the solution to (29) into the N’ equa- 
tions of constraint (4) should then determine the multi- 
pliers uniquely for the final explicit solution to (5) for 
W(r). Then from (3) the actual optimal weighting func- 
tion becomes 

Voie 
oS gC ae 
where g’(7) is equal to g(r), the periodic pulse modulator, 
over the nonzero intervals and is continuous in between 
[discussion before (3)]. 


Parr [1I—Fintrt Memory ConsIDERATIONS 


In Parts I and I], the infinite memory problem was 
solved. It was shown that the general solution was the 
conventional Wiener Solution for the unmodulated case, 
plus a series of appropriately weighted delta-function- 
derivative terms at each end point of the pulsing intervals. 


Finite Memory Considerations 


The preceding parts of this study have been concerned 
with infinite memory filtering. That is, the linear operation 
is performed on the entire past of the input time series. 
Formally this implies an indefinite delay before the optimal 
output is generated as a time-ordered sequence. Practi- 
cally, this imposes a delay commensurate with an effective 
over-all system decay constant. In cases where the opti- 
mum response is necessary only at some preassigned time 
point, and where the delay time is more restricted, the 
problem takes the form of a finite memory optimization. 
Such an optimization of unmodulated time series has been 
carried out in reference [2]. The following is offered as a 
means for utilizing and extending the analysis for un- 
modulated inputs to apply to the periodic pulse-modulation 
case. 

The transition to the finite memory problem is made by 
replacing © as a limit in all integrals by PZ’ (except 
where an ideal operation is denoted). This limits the dura- 


n of operation of the filter to an integral multiple, P, 
the pulsing period, 7; so PT is the memory extent of 
filter. Thus in the subsequent analysis, finite memory 
ations will usually have infinite memory counterparts 
id, for the sake of convenience, use will be made of the 
: responding infinite memory equation number primed. 
rr example, the infinite memory case 


“yf drw(n¢glr’ — 2) — ua le 7 Ste 


ere W(r) = 0, for7 < O and mT < + < mT — h, all 
tegers m, and the finite case becomes 


yf DTW (1)OlT! — 7) — vay} 


= 0, Veal be Ree NY, (9)’ 
ere W(r) = Ofor PT <7 <OandmT <7 < mT —h, 
integers, m. 
he mode of solution in the infinite memory case has 
en first to solve for the unpulsed optimum filter, then 
r showing that the filter for the pulsed input (with the 
dition of sampled output) effectively reduces to a 
| eee device (8) to determine the coefficients of 
D + 1 derivatives of 6-functions at the end points of 
fh pulse. 
‘The same method is to be used in the finite memory 
‘oblem. The optimum filter for the unpulsed input in 
is case is first formally obtained (for mstance by the 
ethod in reference [3]). Then (9)’ requires D + 1 deriva- 
ves of 6-functions at each end of the pulsing intervals, 
at here there are only P pulses instead of infinitely many. 
hus (8) indicates a finite pulsed-memory filter and the 
lowing seems feasible: given WW(r) such that 


IPT 


dwi(ne(r’ — 1) — Wr’) =0, OX 7 SPT (19)’ 


en by utilizing u(—7)W/(r) as the part of the weighting 
nection W(r), which is continuous over the pulse duration 
d including the D + 1 6-function derivatives at the 
dpoints of the P pulse intervals, (20)’ becomes 


RY 


dru(— 7) W6(r)b(r’ — 7) 


tT=mT—h 


ee (ue 25) 


m=1 r=0 


Omer) @ npr >) | 


=r) =f drwilrig(e’ — 9). 


nT —hs7 <n, 


Loe es (20)’ 
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Then (21)’ through (31)’ follow as in the infinite memory 
case. 


De {Qmt—h,r at B(m—1) 7, PKL = h)}(—ax)" 


r 


T-h 


= ¥ wi((m — 11) ‘} we pends 


ys {Qmt—n.r + Onan meee a an h) ag 


r 


= Lvkem = 0) fo we(abels) ar. 


K’ 


,D and m = 1 (31)” 


EXAMPLE 


In this case we consider a purely stationary input 
consisting of random signal plus noise. The desired 
operation is again prediction. Further, let the modulator 
of the input time series be a uniformly-pulsed function 
of the form 


nl <7 < nT +h. 
Woh exe al 


= 1+ asin——, (32) 


The signal is assumed as follows, uncorrelated with the 
noise: 


eae a. ena 


On(T) = 


2 — 2 - 
oe Bir ae oe yirl 


SC) 
11(7) = 


(33) 


where ¢,, and ¢, refer to signal and noise autocorrelation 
functions respectively and ¢,, is the total input autocorre- 
lation function. 

The transform of the corresponding Wiener-Hopf 
integral equation for the unmodulated case (12) becomes 


Sn ea ae Can 
—B\p -—y)/ p— Bw 


rales G(p). (34) 


pd 


is the transform of the ideal predictor, where 


ky 


2(Bomn + Yon), he = A (y°Bon + B’yo,). (85) 


2 
C =a, 


Thus, as in (11), 


vq) = -—Cetae+y ‘e ==) ie. at 
: Bas Pia Phy pe. 
(36) 
or, if we define, 
SS Ea 
cS sees (37) 
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then 


r Cks ris 9) eos Io | « 
= ee Te 38 
Y(p) ks ( . sia D ui k, | ( ) 
and thus 
ye Ck: Bd " -kaT) Q¢ 
Wot) = en OCr) ya ae eet (39) 


The form expressed in (27), 


. Ck. A) —komT —kot 
WeQil Hip iene © He OntoO\ 0) pay ah ee ee 
(40) 
Then let 
Ck. 
ky = ie ; ks = (y = ko) kg. (41) 
1) 
Thus 
wml) = k,6(mT), Wo(r) = d(7) (42) 
CIs ers way ee 
Turther, define 
a T-h e = 
Weer = Wb, (7) dr, 
0 
T-h 
fie = / Coe aes (43) 
0 
Thus, if 
dor) =e", gilt) =e” 
we obtain 
Woo — il. 
Wo1 ee i 
Wi = : z : 1 — efoto (r-m) 
z 
Wi = B = ke 1 Po a ae 
2 
Woo = 1 
Wn = 1, 
Ee 1 __ ,(B+ka) (Th) (44) 
Wio0 B ae ky (1 ) 
wD ~ 1 (1 a. sou eae 
zi oY; ae ky 


Thus the system (31), for the coefficients becomes 
TIEN ed ems FS at d(T — h)dim—1),7,0 


— O(T — A)acdim-1) 7.1 = yo Wok’ 


June 


Gmt—-h.0 — Amr-n,v + OT = h)@en-aya.o 
aT d(T == h)ayim—-1y 7,1 = ye WK’ 
= 


(45) 

Oyen 4.6. Cfo Oo, Le eee 

+ oo (LP — h)aoQ (mar. = pe Dox’ 
Op veho ce Qionrn td fo, lO eee 

+ $1 (0 Moda = 2 Dik 
which has the solution, 
Gim—1y7,0 = —A'{F(eoor($o + $0') — (bi — 41°) 

+ Frlaolbo + $0') — ab. —))} apy 
Ginna = A'{Fy(eal$o — ¢0') — aolbi — 61°) 

+ Frlbo + $0 — $1 — o1')$ 
where, 


A = —ayay((bo + $0") — (61 + ¢1')) (Go + 0° — $1 — or) 
+ (aldo — $o') — auld: — 1 )} 
-(ax(bo — $0') — aol: — $1 )) 

Pio fo fa eee 


BP. re ai(fo a fe) Sei ao(fi = io) (47) 
fi= Diwe, t= 0,1, 2,8, 
and 
QmT—h,0 — ztfo mi fe = (ho aie ho as 
== ao(fo ae bo )as} 


(48) 
OmT—h,l — {fo Se oeaa (bo <T bo ) As 


<6 ao(do =i do )a4} : 


Thus, the optimal weighting function, for a modulator 
of the form u(r)(1 + a@ sin 227/T) is expressible as 


u(—7) 


WG es + asin 2x7/T 


{k46(7) Sie kenrg 


op Xe, % {(—)'Qnr—-n.,5 (4 — mr — h) 
si 


(=a Ot = mr)}}. (49) 
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| Summary—Shot noise, represented by a series of impulses with 
pisscn distribution in time, and with arbitrary time-independent 
nplitude distribution, is sent through an RC integrator. Time- 
pendent statistics of the output are investigated by means of an 
itegro-differential equation describing the statistical flow. The 
kact second-order probability density of the output is obtained. The 
e-dependent Edgeworth series for zero initial output is exhibited, 
jd is seen to bear a simple relation to the familiar equilibrium 
dgeworth series. Results are shown to reduce to those of the 
pkker-Planck equation describing integrated ‘‘white’” noise as the 
equency with which impulses arrive becomes infinite. 


N RC integrating circuit with impulse response e-'/” 
is acted upon by noise impulses, y6(4 — t’), with 
Poisson arrivals in time with mean frequency of ar- 

val 1/7, and with time-independent amplitude proba- 

ality density A(y). Let x be the output of the integrator. 

"he first-order equilibrium density of the output, W (2), 

‘ell-known.’ We derive here the second-order density, 

V2(v1, @2, 7) by solving the time-dependent equation 

escribing statistical flow.” 

| Consider an ensemble of systems with amplitude 

ensity W(x, t). The development of W(x, f) in time will 

‘e described by 


Com) ee (2) he 
pp on \7 me AC Ta 


a aa Wa’, th A(x — 2’) dr’. (1) 


1s 


‘his may be seen by considering the conditional density 
.(v’ | x, A) for an infinitesimal time A. The probability 
f no pulses arriving in time A is (1 — A/7z,), in which 
vent all system amplitudes decay to x’ e *’”. The prob- 
nility of a pulse arriving is A/7,, in which case the systems 
amp from x’ to x with probability A(a — 2’). Hence, 
ra’ | a, A) = b@ — a e&°”)(l — Afr) + A/ro 
(c — x’) to ane order m A. Now W(t + A) = 
ce W(x’, t)P.(a’ | x, A) dx’. Inserting the expression for 
poy ie with respect to A, and taking the limit 
s A — 0, we obtain (1). 

Eq. (1) is readily solved eo one; the I ae transform 
f both sides. Letting ¢(k = [oe Wi, Dude, we 


* Manuscript received by the PGIT, November 17, 1958. 

7 Sylvania Electronic Systems, Waltham, Mass. 

18. O. Rice, “Mathematical analysis of random noise,’’ in 

'. Wax, ed., “Selected Papers on Noise and Stochastic Processes,’’ 

over Publications, Inc., New York, 154, eq. (1.5-4); N. Y., 1954. 
2 J. E. Moyal, J. Roy. Stat. Soc. B, vol. 11, p. 190; 1949. Using 

1e Langevin Equation, Moyal discusses the time- dependent sta- 

stics of Poisson noise in an RC integrator in the case where all 

ulses have the same amplitude. Thus, the results of the present 

iscussion reduce to his in the case A(y) = 6(y — Yo). 
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or 


he Second-Order Distribution of Integrated Shot Noise” 
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obtain 


dg(k, t) 


Alon eae pa ea a 


To 


—— lent), 2) 
where @(k) is the Fourier transform of A: @(k) = J2. € 
A(y) dy. Letting 6(k, t) = e*“’?, we transform (2) into 


oy , kav @&) —1 
Ot r Ok Tq (3) 


The general solution to (3) is given by any particular 
solution plus the general solution to the homogeneous 
equation: 


+S = 0. (4) 


Physically we expect that an equilibrium distribution will 
be reached, so as a particular solution to (8) we choose 
¥.q(k), a function independent of ¢. Eq. (3) can then 
immediately be integrated to give 


Gk) I 
Welk) — WPeq(0) — zf ( 1, dk’ fe 

Since 

1=f. Wa, t) dx = 90,1) =e", 
YO, t) = 0, and in particular y.,(0) = 0. Hence, 

@(k’) — 1 
Yadt) = = f SORT ay (5) 
(0) vu : 


The general solution to (4) is obtained by letting u = 
log k, whence (4) becomes 


oy , low _ 
AOL a ay, ’ 


which has as its general solution an arbitrary function of 
u — t/r = logk — t/t = log (ke ‘’’). Thus, the general 
V(k, t) = F(ke ‘’") + .,(k) where F is determined by the 
boundary condition W(k, 0) = Yolk), Wo(k) m turn 
being determined by an arbitrary initial x distribution: 
W(a, 0) = W(x). Setting ¢ = 0 in the general solution, 
we have ¥(k, 0) = ¥.,(k) + F(k), so that F(k) = Wo(k) — 
y.,(k). Thus, the general solution to (8) is 


Hk, ) = Youll) + Volker”) = Youle”). ©) 
Since o(k, ft) =e”, 


exp [Weag() | exp [Wolke@'”)]_ 
exp [Weg(ke ‘’")] 


o(k, t) = 


That is, 
Pealk) 


cemiaa\ 
9..(ke-'”) poke °”") (7) 


ok, t) = 
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From (5), 


p(k, t) = exp ‘Z i re je =| an’ bate) 


If we let k’ = ke “’’ this becomes 


$(k, t) = glk, tpo(ke””) (8) 


where 


Gh; 2) "= 3Exp ‘A fi [@(ke“’") — 1] inh. (9) 


We now examine our solution in coordinate space. We 
obtain P,(a» | x, t) by letting W,(a) = 6(@ — 2). Then 
do(k) = e**** so that (ke ‘’”) is the Fourier transform 
of 6(« — x e ‘’"). If G(a, t) is the Fourier transform of 
g(k, t), G(x, t) = 1/2mr f%. e™* g(k, #) dk, then we infer 
from (8) that P2(a, | x, ), the transform of ¢(k, #), is the 
convolution of G(a, f) and 6(a@ — aye‘’’); @e., 


Pita.) Gira cee (10) 
Given an arbitrary initial density W (2), 
We t= i Wilao)Po@e | 2.4) dt, 
so that 
WG, 4) = ie W o(ao)G(a — xe”, t) dao. (11) 


As t — o&, we see from (11) that W(a, ¢) approaches a 
limit independent of the initial W,(a 9); 7.e., there is a 
unique equilibrium density, 


W.a(x) = G(x, ©). (12) 


We display W..,(7) explicitly in terms of A by taking 
the Fourier transform of (9) and expressing @(k) in terms 
of its Fourier transform A (x): 


Wola) = 5 [dhe 


“exp if au | dy A(y)[exp (kye“’”) — uh (12a) 


To 


This is in agreement with the known result for W.,(a).” 


The second-order density is now given by W,,(%o) 
TERS Giga mie (aN mas 


W Go,:0) 0) = Wig 2) Gia te2 (18) 


In addition to the existence of a unique W,,(x), we can 
verify some other properties of the solution that one would 
expect to hold. That P(x, | x, ft) is always positive follows 
from the fact that it is a solution to (1) which is initially 
positive, for let x’ be the earliest point at which P, becomes 
negative, and let P:(x, | 2’, t) pass through zero at ¢ = ?’. 
Then P2(x | x’, t’) = 0, whereas P2(x, | 2, t') > 0 for z 
in the neighborhood of 2’, and hence 


a ee ° 
aun = all Bae Pelee | «, t’) 


z=x' z=x! 


Jun 
These facts, however, along with (1) require 


0 
des ’ = 
aj Poo | 2’, t) ee 


, 


which contradicts the assumption that P, becomes nega 
tive at x’ an instant after 7’. 

For the solution to describe a Markoff process, we mus 
have P3(ao | a, f) =" (25 dz” Poot oes) ts eee 
Taking the Fourier transform of both sides and notin; 
(10) and (9), we find this condition equivalent to g(k, ¢) = 
g(k, t — s)g(ke “‘~°”, s), which is in turn readily verifies 
from (9). 

We now obtain the time-dependent generalization of th 
equilibrium Edgeworth series, given that systems hav 
amplitude 0 at t = 0. In this case, W(x, t) = P2(0 | 2, t) = 
G(a, t). The semi-invariants, \,, of G(x, #) are defined u 
terms of its Fourier transform g(k, t) by 

wis) = exp 4, 


Since 


ny nr 


See ees 
ak) = di nl 


n=0 


where y" =[ y" A(y) dy, 


(9) may be written 


gk, t) = exp iz es 


o an nn! 


iy"(1 ax erin} 
from which we conclude 


m= FE ae = nore), C4 
where X,,(0) are the semi-invariants of the equilibriun 
distribution. 

If we are interested in the distribution about the averag 
[¢ = A.) = (r/o) GA — e'’")] normalized to uni 
mean square at / = ©, we must investigate the charac 
teristic function gy(k) of the distribution in 


A simple transformation gives 
k’ — iby Tt — . n nN t 
gw(k, t) = exp ‘5 (d—e?7") + = (tk) mot 
n=3 . 


where the semi-invariants, \/, of the normalized distribu 
tion are given by 


M = 0 
ee = 1 Ty, Cree 
y % (2) 26 y" (1 hoe a) (14 
> n TD pa\ ns 2 € 
(y) 


= Cc )( Le Ceetean 


S RICE Op Cite Lote 


we collect terms of the same order in (7)/7), we obtain 
» time-dependent Edgeworth series for W(Z, ?), 


y Meg a | 
Wf = i ee 3 : 4 
ai ax © Ee 


per hues 


(16) 


The df are just the coefficients of the well-known 
ulibrium Edgeworth series with time-dependent 
‘tors (1 — e"'’"), and the equilibrium function 


= 29/2 
e 


Vr 


ummary—Two channels carry noise waveforms, No(t) + Ni(t) 
N,(t) + No2(t), where N,(t) is a common narrow-band Gaussian 
se and N,(t) and N,(f) are independent narrow-band Gaussian 
es associated with each channel. The outputs of each channel 
_sent through detectors whose outputs, F(x, y), are identical 
ogeneous functions of the components, x and y, of their inputs 
where N(t) = x(t) cos wot + y(t) Sin wot. Let Ri2(r) be the 
malized cross-correlation function of the two detector outputs. 
s shown that to determine R,.(7) it suffices to know the normal- 
d auto-correlation function Ro(r) of the output of a single such 
ector when the input is No(f); i.e., if Ro(r) = G(oo?, p(r)) where 
and oc, are the normalized auto-correlation function and rms 
either component of No, then it is shown that! Rio(r) = G(o?, 
)) where Z = [{1 + (01?/o02)} {(1 + (02/00?) }}7?. 


ET narrow-band Gaussian noise be represented by 
N(t) = x(d) cos wot + y(t) sin wot, where x(t) and 
y(t) are independent Gaussian noise processes with 

mtical distributions. We say that a detector is homo- 

neous if its output at time /, F,, is a homogeneous 
yction of order y of the components of its narrow-band 

ise Input; 7.e., 


F, = F(x(t), y@) (1) 


FOAL, YY) = NEE; y) torall Xx. (2) 


* Manuscript received by the PGIT, November 17, 1958. 
Sylvania Electronic Systems, Waltham, Mass. ; 
Several previous papers discuss the cross-correlation function 
aussian processes after nonlinear no-memory transformation. 
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is here replaced by 
BOUIN: 
V anri(t) 

Note that as 7, — 0, (16) goes into the Gaussian solution 
of the Fokker-Planck equation for integrated white noise. 
It is also readily verified from (13), (12), and (16) that 
the appropriately-normalized joint probability density 
W (ao, x, £) becomes bivariate Gaussian as tr) — 0. For the 
conclusion of (16), that the normalized G(x, t) becomes 
gaussian as tT, — O can immediately be generalized to 
G(x — ae ’’”, t). Since W(ao, x, t) = G(x, -) G(x — a 
e ‘’’ t), the Gaussian character of W in the limit 7) > 0 
follows. 


A Theorem on Cross Correlation 
Between Noisy Channels’ 


J. KEILSON], N. D. MERMIN{ anp P. BELLOT 


Such a detector might be, for example, a square law 
detector, with F = x + y’(v = 2); an envelope detector, 
with F = V2? + y (v = 1); or a phase detector, with 
F=2/V2+y ( = 0). 


The normalized auto-correlation function of F will 


depend only on the rms of either component, ¢ = V 32 = 


Vy, and the normalized auto-correlation function of 
either component: 


p(t) = a(da(t + 7)/o” = y(Dy(t + 7)/0’, 
ces 


EE 


iG) = = 
(7) F? on yi 


= Go’, p(7)). (3) 


Consider now two channels having identical homo- 
geneous detectors with noise inputs Ny + N, in channel | 
and N, + N, in channel 2, NV, and N, being independent 
noise waveforms associated with channels 1 and 2 respec- 
tively, and No, a third independent noise waveform 
common to both channels. No, N,, and N; are all narrow- 
band Gaussian noise. NV ;(t) = «;(f) cos wot + y;(¢) sin wot, 
7 = 0, 1, 2. Let (/,), be the output of the first detector, 


(Fi), = F(xol(t) + ai(0), yo) + y:(d)). (4) 
Similarly, the output of the second detector is 
(Fs). = F (x(t) =— s(t)’, Yo(t) ae y2(t)). (5) 
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It is sometimes desirable to compare the two channels by 
cross-correlating the normalized detector outputs rather 
than the narrow-band signals themselves. The normalized 
cross-correlation function of the output of the two detectors 
is 
i) (Po)ive = Mi Ms 


R,o(7) = (6) 


Vit = FU Fe) 

It is our purpose to show that if the normalized auto- 
correlation function of a single such device is known as 
a function, G(o’, p(r)), of the normalized auto-correlation 
function of the input noise components, then R,, may be 
obtained simply in terms of G, 7.e., the following theorem 
holds. 

For homogeneous F, 


R,2(7) = G(o, Zp(7)), (7) 
where p is the normalized auto-correlation function of 
either component of No; G(os, p) is the normalized auto- 
correlation function of a single homogeneous device F 
with input No, as defined in (3). 

And 


(8) 


where 


a2 ae 
OR = Us 


= 
To prove the theorem we use two well-known properties 
of the Gaussian distribution: 
1) An nm dimensional Gaussian distribution (with zero 
mean), W(X, - X,,), 1s uniquely determined by 
its n(n + 1) second moments, X;,X; . 
2) The sum of two Gaussian variables is itself a Gaus- 
sian variable. 
Because of 1), if X, ---X, are any four Gaussian variables, 
then 


PKG) OE) 


i ie | F(X, X2)F(X5, X4) 


“Wi Xa XG) dky e = aX, 


Orn XG, ye See Nee 


OCP. OP. CFS GOD. ai © AO) 


where gp,p is a function of the second moments completely 
determined by F. 
[hi XG, SS ae, Sa NGS hee wil) 


X4 = Ys = Yo(t oF T) 


Xs 


== 


then 


Xe ve 4 is Se OG, ae X,X4 = oop(7) 


INFORMATION THEORY Jun 
and all other moments are zero. Hence from (9) 
PP a= Fae meee 

= grr(oo, Oy %0, Fo; On Goe(T) Op 0 copa) mae)) 
ga(oo, plz). 
Now let 


(10) 


/ 


X, =a(% +a), Xz = a + y), 
X3 = an(vo + a), Xs = aa(ys + Yo) 
where 
a, = [+ i/o) ne 


Because of 2), X, --+ X,4 are Gaussian variables so that 
(9) may be used. The moments are 


NE OC Ga = G 
E.G AG Pu) 


2 3 : 2 5 2 
X,X3 a XX = Aa2p00 = Z poo; 


Xe 
al 


and all other moments are zero. Hence 

Fle@i(ao + 21), on(Yo + y:))F(ee2(@o + 22), a2(Yo + Y2)) 

Yr r(F, 0, 0, 7, 0, Zpao, 0, 0, Zpao, O) 

gas(oo, Zp(7)). (1g 
Due to the homogeneity of /’, the left side of (11) is equal te 


I 


aiask (x, ar Gio Uy SF YF (aé == dla. Yi =e Ve) 
= Z" (F,) (Po)e+m 


Thus 


Ta Cyl eas tea : 

(Fi) (Pe) = my gay(oo , Zp(7)). (12) 
To obtain the remaining moments that make up R(7) 

and I?,.(7), we make use of similar considerations in two 

dimensions, 7.e., if H(X,, X,.) is a function of two Gaussian 

variables then 


[| HG, 3) ax de, a eee 


If X, = 1, Xo = Yo, then X*% = X7 = 0, x ee 


Hence 

HGe Go) = dale aan): (13) 
When X; = a;(a) + 2); X. = aio + Y,), 2 = tye 
then Xs = \0— "0 eee 

Hela + 2), a(Yo + Ys) = guloo, 09, 0). 

If H is homogeneous of degree p, 
a; H(t + %:, Yo + Ys) = Guloo, 70,0), = 1,2. C4 
If H = F, then » = v and (18) gives 


PF = Jr(oo, Os Oe Gca)(o0)- 


. (14) gives 


1 2 i 
=e Jr(ao » Io , 0) Soe 9i2)(o9), B= Ue Py: (16) 


O i 


ee if H = F’, » = 2y and (13) gives 


F* = gro, 95, 0) = ge(o). (17) 
y. (14) gives 
i Jr2(o9, 0, 0) = sh) gis(oo), @= 1, 2. (18) 
Q; Qa; 


‘As a result of (10), (15), and (17), (3) can be written 


R )= = Gaslao, p(7)) oa (Gi2(o0))”_ 19 

(7 9.3)(oo) — (912)(o0))” (19) 
(On the other hand, placing the results of (12), (16), and 
) in (6), we find that factors of 1/aja} = 1/Z” appear 


Summary—Signal-to-noise ratios associated with smooth band- 
BS limiting and subsequent narrow-band filtering of a periodic 
‘nal and random noise are computed. Observed changes in 
‘nal-to-noise ratios may be used to estimate detectability losses. 
.e error function is used to represent the limiter characteristic 
various degrees of limiting. First-order corrections with an 
reasing input signal to the signal-to-noise ratios, which are 
sed on the small signal theory, are computed for limiter input 
se with sin x/x, Gaussian, and exponential correlation functions. 


INTRODUCTION 


IMITING or clipping of noise has been studied 
by several authors.’ ° Hard-limiting of a signal 
plus noise has been investigated as a limiting case 


* Manuscript received by the PGIT, October 16, 1958. 

+ Appl. Res. Lab., Sylvania Electric Products ‘Inc., Waltham, 

ASS. 

1 J. H. Van Vleck, “The Spectrum of Clipped Noise,’’ Radio 

is. Lab. Rep. No. 51; July 21, 1943. See also J. L. Lawson and 

_E. Uhlenbeck, ‘‘Threshold Signals,’ M. I. T. Series, MeGraw- 
1 Book Co., Inc. New York, N. Y., vol. 24, pp. 57-58; 1950. 

2D. Middleton, “The response of biased, saturated linear, and 
dratic rectifiers to random noise,” J. Appl. Phys., vol. 17, pp. 
-801; October, 1946. 

Jae Bussgang, “Crosscorrelation Functions of Amplitude- 

storted Gaussian Signals,’’ M.I.T. Res. Lab. of Electronics, Tech. 

p. No. 216; March 26, 1952. 

 damlsl. Laning and R. H. Battin, “Random Processes in Auto- 

itic Control,’’ McGraw-Hill Book Co., New York, N. Y., section 

L; 1956. 

oR. F. Baum, “The correlation function of smoothly limited 

Jussian noise,”’ TRE Trans. on INrorMATION Tueory, vol. IT-3, 
BOB =1975 September, 1957. 

6S. A, McFadden, “The correlation function of a sine wave plus 

ise after extreme clippings,’ IRE Trans. oN INFORMATION 

meORY, vol. IT-2, pp. 82-83; June, 1956. 
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in both numerator and denominator and cancel to give 
gay(oo, Ze(z)) — (Gi (oo) 
2 
9<3)(o0) — (gca(a0))” 


A comparison of (19) and (20) now reveals the validity 
of Theorem 7. 


Ry(7) = (20) 
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Signal-to-Noise Ratios in Smooth Limiters* 


JANIS GALEJSt 


of a symmetrical n-th root circuit with n approaching 
infinity.” Such an operation degrades the output signal- 
to-noise ratio of a band-pass limiter by a factor of (4/7) 
relative to the linear case for small input signal-to-noise 
ratios. The deterioration of signal detectability by similar 
band-pass limiters has been determined for several spectral 
shapes of the limiter input noise.” The deterioration in 
signal detectability ranges from approximately 6 to 16 
per cent. The above results suggest that only small changes 
in the output signal-to-noise ratio in a narrow-band filter 
following the wide band-pass limiter can be expected as 
the limiter action is varied from a linear characteristic 
through various degrees of lmiting to hard-limiting 
referred to above.’'’* However, in systems employing a 
multitude of parallel band-pass filters, a small per cent 
difference of signal-to-noise ratio in each of the channels 
may result in significant changes of the over-all false 
alarm rate or incorrect dismissal probability. The signifi- 
cance of determining the exact amount of signal-to-noise 
deterioration by various saturation or limiting levels of 
the filters at different input signal-to-noise ratios is 
apparent in such systems applications. 

The narrow-band filter which follows the limiter restores 
to a large extent the Gaussian character of the noise 


7™W. B. Davenport Jr., “Signal-to-noise ratios in band-pass 
limiters,” J. Appl. Phys., vol. 24, pp. 720-727; June, 1953. 

8 R. Manasse, R. Price, and R. M. Lerner, “Loss of signal de- 
tectability in band-pass limiters,’ IRE Trans. on INroRMATION 
TuHeory, vol. IT-4, pp. 34-88; March, 1958. 
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: : : ; sees 9,10 ; 
which is lost in the wide-band limiter outputs. This 


results from the averaging action of the narrow-band 
filter and is a consequence of the central limit theorem.” 
The computed signal-to-noise ratio when used in conjunc- 
tion with a Gaussian amplitude distribution gives a first- 
order solution for changes in the false alarm rates and 
incorrect dismissal probabilities in a subsequent threshold 
device. A more accurate higher-order solution is obtainable 
by better approximations to the actual probability distri- 
bution in the narrow-banded limiter output. Although 
the determination of the exact probability distribution of 
the narrow-band filter is a rather complex problem, 
approximations of various degrees to it can be worked out 
by applying Edgeworth series.""'” 

The specific system considered in this paper consists of 
a limiter followed by a narrow-band filter, as shown in 
Tig. 1. The waveform e;(f) is wide-band noise plus a 


i €o (t) 
elat)) NARROW BAND co) 
| 2 
LIMITER FILTER 


Fig. 1—Block diagram of the limiter circuit. 


ej (t) 


sinusoidal signal. The limiting function relating the 
limiter output to its input 


e’(t) = Lle,(d)] 


is selected such that it: 1) approximates the operational 
characteristics of physically realizable limiters with a 
linear small signal behavior and with a gradual saturation 
and, 2) yields readily integrable expressions in the subse- 
quent analysis. The error function 


Oe va [ FR a: (2) 


satisfies both requirements. With 


2 =) a 


the parameter ‘a’ determines the limiting level as indicated 
in Fig. 2. It is seen that for ‘a small, L(z) approximates 
the characteristic of a hard limiter. For ‘a’ large, L(z) 
acts as a linear device of gain 


n= 


9A. J. F. Siegert, “Passage of stationary processes through 
linear and non-linear devices,’ IRE Trans. on INFORMATION 
Tueory, vol. IT-3, pp. 4-25; March, 1954. 

10 B. Gold and G. O. Young, “The response of linear systems to 
non-Gaussian noise,’ IRE Trans. oN INrormation Tuuory, vol. 
IT-3, pp. 63-67; March, 1954. 

1 Jj. KH. Storer and J. Galejs, ‘Effect of Limiting on the False 
Alarm Rate and Signal-to-Noise Ratio of an Envelope Detector,” 
Sylvania Electric Products Inc., Waltham, Mass., Appl. Res. Memo. 
No. 135; June 9, 1958. 

2 P. Bello and W. Higgins, “Effect of Limiting on the Proba- 
bility of Incorrect Dismissal at the Output of an Envelope De- 
tector,’ Sylvania Electric Products Inc., Waltham, Mass., Appl. 
Res. Memo. No. 163; March, 1959. 
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Fig. 2—Limiter characteristic. 


This shape of the limiting function is the only one known 
that gives for a noise-only input a simple closed-form 
expression for the autocorrelation function of the limiter 
output at various degrees of limiting.” 


AUTOCORRELATION FUNCTION OF THE 
Limiter Output 


The characteristic function method of Rice’’ is used for 
computing the autocorrelation function of the limiter 
output. The autocorrelation function of the limiter output 


1s 


R(x) = im i i dt iP du L(u) exp (—0.50°u? 
f dy L(v) exp [—0.50°v” — o’p(r)w] 
-exp [juS(t) + wS(é + 7)], (5) 
where 
Li) = [Elo exp (jen) du 6) 


S(t) is the signal part of e;(¢) 
a is the rms value of input noise and 
p(r) 1s its normalized autocorrelation function. 


For the limiting function L(z) of (3), the Fourier transform 
is given by’ 


L(u) = exp (—aw)/(ju). (7) 


Assuming a sinusoidal signal and computing the time 
average of (5) prior to evaluating the integrals over 
and v gives 


8S. O. Rice, “Mathematical analysis of random noise,’’ Bel 
Sys. Tech. J., vol. 23, pp. 282-332, 1944; and vol. 24, pp. 46-156. 
sec. 4.8, 1945. | 

4G. A. Campbell and R. M. Foster, ‘Fourier Integrals for Practi- 
cal Applications,” D. Van Nostrand Co., New York, N. Y.; 1948 
See also, A. Erdelyi, W. Magnus, F. Oberhettinger, and F. G 
Tricomi, ““Tables of Integral Transforms,’’ McGraw-Hill Book Co. 
New York, N. Y., vol. I; 1954. 

This pair of Fourier transforms is depicted in Fig. 4, p. 15 anc 
can be derived from the transform pair 725.1, p. 86 of Campbel 
and Foster, or from the relation (2.4.21), p. 73 of Erdelyi, et al. 


i du L(u) exp (—0.507w 


[ dv L(v) exp (—0.50°0" 


-exp (—a'p(r)w) J o(S Vu? + v? + 2uv cos aor). (8) 


‘Series expansion of the expression in the square brackets 
} 15 


RG) = aS; ie Gil hike cos met (9) 
ere 
Ine = kat / L(u) u'J,(Su) exp (—0.507u”) du — (10) 
= 2 ae Oars (11) 
abstituting (7) in (10), defining 
| b=a+ 0.50" (12) 
ad 
as s? RO) 
pombe 4G 0 Se. Ae a” Ue 
1e evaluation of the integral gives’” 
| —0.5k. 0.5n 
ab? w = FEE yn + 15 a0) 
; (? —k— ") 2 
ae ca ee 
= 4 (14) 
forn + k odd 
LO forn + k even 


hich reduces for hard limiting (a = 0) to the result of 
avenport.’ The fundamental component of the signal 
= 1 term. Substituting h,> in (9), 


T W 


2 
Raa) = jv (0.5; 2; — w | COS WoT. 


(15) 


he noise terms of the correlation function are all the 
rms with k # 0. Substituting (14) in (9) 


Sy ol ab *w” 


ree el 2 
n=0 k=1 far (? : "| 


(n+k=odd) 
|e ieee é 

all alia so, ;n — w] | €, COS NwoT. 
ith the parameter ‘a’ always larger than zero, w of (13) 
ll be less than the input signal-to-noise power ratio. 
pr small signal-to-noise ratios, one may ignore the higher 
wers of w. Considering only the two lowest power terms 
w, (15) and (16) simplify to 


R%(r) & 2rw(l — 0.5w) cos wor 


fe (7) = 


ag als (16) 


(17) 


15 See sec. 4.9 of Rice, op. cit. 
16 See (4.10-5) of Rice, ibid., or the development of Davenport, 
B ihe 
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and 
ME BSE IE a...) 
Ria) 2n(W + 4 3 + 15 + .:- 
+ mo] —(w + hws + 23 we 4 ) 
wl We +3 wet ) cos aur | (18) 
where 
ei a p(T) sqett) COS @1T 
Me eT o + 2a a 


The first summation of (18) is recognized as the sin’ W 

function.’ An expression analogous to the autocorrelation 

function of McFadden® is obtained by neglecting terms 

containing w’ and higher powers in (16) and by letting 
= 0 (hard limiting). 


OurpuT oF THE NARROW-BAND FILTER 


The autocorrelation function of the filter output is 


given by the convolution 
Ri(z) = R(r) * 5" | Z@) |? (20) 


where Z(w) is the transfer function of the narrow-band 
filter. Letting 


Rr) = ¥* | Z@) |, (21) 
the mean-square filter output is simply 
RO) = [  R@R.(a) ar. (22) 


A single tuned RLC filter is assumed for the narrow- 
band filter."’ In a high Q narrow-band filter, its natural 
frequency is much larger than the filter half-power band- 
width 


= 2d. (23) 


The filter impulse response and also the function R,(r7) 
may be approximated by ** 


R.{7) 

The mean-square signal output of the narrow-band 
filter is computed as indicated in (22) with R,(r) given by 
(24) and the autocorrelation function of the limiter signal 


output R?(r) given by (17). Lining up the natural fre- 
quency of the bandpass filter with the signal frequency, 


(25) 


~ K exp (—d| 7 |) cosa, | 7 |. (24) 


We = Wo- 


17 Although a specific filter characteristic is assumed for the sub- 
sequent analysis, the final results, that is the degradation of the 
filter output signal-to-noise ratio due to limiting, are independent 
of the filter shape or the filter-bandwidth, provided that the filter 
bandwidth is much less than the bandwidth of the input noise 
spectrum. A straightforward computation of signal energy and of 
ihe noise spectral density at the limiter output in the vicinity of 
the signal frequency would yield identical results. 

18 The relation (24) can be derived with the aid of (448.1) and 
(449.1) of Campbell and Foster, op. cit. 
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Applying (25), the signal power output of the narrow-band 


filter becomes 


Sy = 4rkw(1 — 0.5w) | COS woa exp (—dx) dx 
iD (26) 
2rikw 
= =~ (1 — 0.5w). 
d 
Substituting (26) in (22), the noise output power of the 
narrow-band filter becomes 
Ne 


2h [ie cos wt exp (—dx)R*(x) dx. (27) 


Jo 
The autocorrelation function of noise as given by (18) 
three summations. Expanding the odd W 
powers of the first and second summations, only those 
terms proportional to cos @,7 will give rise to noise com- 
ponents within the filter pass band and will contribute 
significantly to the integral (27). The amplitude of these 
terms is obtained from the relation 


ate ales + ') 
COs =F COS Y 


m 


consists of 


aa) - et 
(a? _ 1) ©% By + : (28) 


The third summation contains only even powers of W 
and of cos @,7. Only the constant part and the part propor- 
tional to cos 2,7 of the expansion of the even-powered 
terms of COs @,7 Will contribute significantly to the integral 
(27). The amplitude of these terms is obtained from the 
relation 


Sere lal aes a) o( 2m ) wee | 
CO — ee ie +2 ibe cos 2y + (29) 


Substituting (18) in (27) and applying (28) and (29) 
results in" 


Poe) 


Ny = Dale | dx exp (—d2) 
Bae ep aed ee OD. | 
“608 (wp — @)a es bes Bos oy ei 
- (B(x) ]?"”? cae 2 [Cn COS (Wo — w1)x 
+ D,, cos 2(w) — alone} (30) 
where 
. en 1 ( o ee 
An = | em m+ 1 \o° + 2a oD 
B,, = (2m - 1)As, (382) 


19 The A,, terms of (30) 
of Manasse, ef al., op. cit. 


are identical with the summation in (11) 


Jur 

Cl Gree os, 2a (3% 

De ma mA m sn (34 
o 


In the above integral, (7) represents the envelope « 
the normalized noise correlation function p(r), 7.¢., 


PT ee 
The integrals of (30) are evaluated in the Appendix fc 


B(7) COS wT. (Sa 


exponential, sin «/x2 and Gaussian functions ®(7). Afte 
evaluating the summations, the noise power outpv 
reduces to the form 
Cy eae _R.O)_ 
big 92a my, 


In the absence of signal or for very small input signal-tc 
noise ratios, the part of the noise proportional to FR, (C 
can be neglected, and c may be set equal to zero in (36 

The output signal power-to-noise power ratio is obtaine 
from (26) and (36) as 


R, LO 


Od a 


> (0:0 i tac) 
Eq. (37) represents the signal-to-noise ratio if noise 1 
measured during signal presence. If noise is measured 1 
the absence of signal, this signal-to-noise ratio is obtaine 
by letting c = 0 in (87). 

The output signal-to-noise ratio depends on the shape ¢ 
the input noise spectrum, on the relative position |wy — w 
of the signal filter within the noise band, and on th 
limiting or clipping level. 

The limiting level is defined as the limiter input amplk 
tude for which the limiter output reaches 90 per cent ¢ 
its maximum. Introducing a proportionality constant 
between this limiting level L and the rms value of th 
limiter input noise oa, 


(= iki (38 
It follows from the limiter characteristic (3) that 
= 33Va. (36 


Designating the value of the constant k for a linea 
system (a and qg infinite) by ko, 


pe en. (4¢ 


Solving (88) and (39) for ‘a’, substituting the expressio 
for ‘@ in (37) and introducing the notation 


i Ol Coe 
(1 + 0.1849@7)ko ’ 


D= (41 
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Fig. 3—Degradation constant, ky. 


he output signal power-to-noise power ratio becomes 


S 
k,| 1 aS =] 42 
ie Lim | - Wr N No Lim ( ) 


Gee 
N No Lim aia: Wa N in 


Se yt 0) 
(3). = o es 


(43) 


nd 


‘he plots of the constants k, and D are shown in I’igs. 3-5 
or various limiting levels g and for noise of exponential, 
in «/x, and Gaussian correlation functions. Noise of such 
orrelation functions is obtained by passing white noise 
hrough single tuned RLC filters, rectangular or Gaussian 
and-pass filters, respectively. The plots are given for 
Y, — W&| = 0 and 0.5a,. The values of the constant ky 
re listed in Table I. It should be noted that the input 
ignal-to-noise ratio of (44) was defined at the input to 
he limiter and that ky of Table I does not take into con- 
ideration the possible attenuation of the signal by skirts 
f the RF or IF amplifier. 
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Fig. 4—First order correction to the output signal-to-noise ratio 
(noise measured in absence of signal). 
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Fig. 5—First order correction to the output signal-to-noise ratio 
(noise measured in presence of signal). 
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soaeacean The signal-to-noise output ratios computed for ver 

ee Biba oabs ie E é small input signal-to-noise ratios require corrections witl 
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DISCUSSION 


The output signal-to-noise ratios of a smooth limiter 
followed by a narrow-band filter were related to the output 
signal-to-noise ratios of a corresponding linear system. 
In the latter system, a linear amplifier is substituted for 
the limiter, leaving the other system parameters unaltered. 
The relative decrease in the output signal-to-noise ratio 
due to limiting, although computed for a specific filter 
characteristic, 1s numerically equal to the decrease of 
signal energy to noise power density ratio, which charac- 
terizes detectability of known signals in presence of white 
Gaussian noise. However, it should be remembered that 
the filter output is only approximately Gaussian. While 
the central limit theorem applies to the noise times noise 
terms of the limiter output, the signal times noise terms 
may yield non-Gaussian statistics no matter how narrow 
the output filter is made. If Gaussian statistics are used 
in conjunction with the computed signal-to-noise ratios 
to estimate losses in signal detectability, such a procedure 
will become less accurate with increasing signal-to-noise 
ratios. It is generally advisable to use the Edgeworth 
series approach to obtain results of higher accuracy.’’'”” 

For very small input signal-to-noise ratios, the degrada- 
tion of the limiter output signal-to-noise ratio is character- 
ized by a single constant k,.”” This constant has numerical 
values within the range of 0.84 to 1.0. The output signal-to- 
noise ratio in the small signal case is deteriorated by no 
more than 16 per cent. Also, the deterioration of the output 
signal-to-noise ratio by the limiter action is more pro- 
nounced at the edge of the input noise band than at its 
center. Manasse, et al.,° observed that the signal detect- 
ability loss is less near the band edge of the rectangular 
input noise spectrum relative to the detectability loss near 
the center of the band. This conclusion cannot be gener- 
alized to hold at the band edge of the rectangular spectrum 
or for the other spectra considered in this paper.”* 


20 The degradation factor of Manasse, ef al., op. cit., is equal to 
the reciprocal of the constant ki, if w Kw, andg = 0. 

1 For a signal located near the inside edge of the rectangular 
input spectrum, the loss in signal detectability is less than it is for 
a signal located in the center of the band as long as the separation 
of the filter frequency wo from the band edge at frequencies 
(w: -+ 0.5 w,) exceeds the filter bandwidth w,. This is not the case 
when the filter approaches the band edge very closely, or when it 
is centered around the band edge where noise spectrum is discon- 
tinuous. That part of the limiter output noise spectrum which lies 
beyond the edges of the input noise spectrum contributes to the 
filter noise output. The noise output of such a filter becomes higher 
relative to the linear case than the output of the filter located at 
the center of the noise band, which implies a larger loss of detect- 
ability. This can be checked numerically by sliding a narrow band 
filter along the noise spectrum plotted in Fig. 3.10 of Lawson and 
Uhlenbeck, op. cit. 


above the rms noise level. The absolute value of D anc 
thus the relative deterioration in the output signal-to 
noise ratio with increasing input signal-to-noise ratio 1 
more pronounced at the center of the input noise banc 
than at its edge. 

The proportionality constant, D, is positive if noise 1 
measured during signal presence. The output signal-to 
noise ratio is therefore improved relative to the smal 
input signal case, which agrees with the conclusion 0 
Davenport.’ The constant D has numerical values of les 
than 0.5, but it is less than 0.16 for the limiting level 
above the rms noise level. The constant D, and thus th 
relative improvement in the output signal-to-noise rati 
with increasing input signal-to-noise ratio, is larger a 
the edge of the input noise band than at its center. 


APPENDIX 


Norse PowrErR OuTPUT OF THE 
NARROw-BAND FILTER 


Noise with an Exponential Autocorrelation Function 


The output of a single tuned RLC filter of half-powe 
bandwidth w, = 2b has an autocorrelation function o 
the form indicated in (35) with 


&(t) = exp (—b | 7 |). (45 


Substituting (45) in (80), the integrals to be evaluated ar 
of the form 


f(t = | cos opener ee 


re ae + (=) J (46 


assuming d < b. 
Applying (46) to (30), 


N,= 2nk| Ee a ae 
fom + nf. + (ess) 
mp PO. pS cxf 2mil 1 + ese) |) 


0) + = = des)" ms a 
oe picamge p> D.2ma 1 + amb : (47 


1959 


Nowe with a sin x/x Autocorrelation Function 


Noise of constant spectral intensity of bandwidth 
= 2b has the autocorrelation of the form indicated in 
(85) with 


(48) 


pSubstituting (48) in (80), the integrals to be evaluated are 


‘of the form 


CG) = / exp (—dz) cos wx (sn be dx. (49) 
0 

\Such integrals are tabulated for n < 3 only.” Integrals 

with larger values of n can be evaluated by repeated 

‘integration by parts. Restricting the consideration to the 

‘cases where d < b, the evaluation is simplified and the 

above integral can be approximated by 


n bx 
L,(w) & ‘E cos we = y dx. (50) 
|The per cent error is less than (100 d/b) for n = 1. The 
‘error is significantly less for higher values of n. 
Evaluating the integral” 
Tv 
yh for w <b 
I,@) = ) a = (51) 
4b for) ie. =D 
O Sion ob 
and 
n-1 
ene +n = 2r) 
es i" O<r<(w/b+n)/2 pian —= 7)! Oo 
with 2 = 2,3, --:. 


| 2 TD). Bierens DeHaan, “Nouvelles Tables d’Integrales Definies,”’ 
Hafner Publ. Co., New York, N. Y., tables 365, 368, and 370; N. Y., 

| 1957. 

23 See pp. 18-20 of Erdelyi, et al., op. cit. 
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Using the above notation for the integrals, (30) can be 
rewritten as 


Ni = oak > _ - 


m=0 


(0) 
+ 20" 


AL a 1 C ml am(Wo aH @) 


» [Fans a @) 


+ Drlom(2eo — 2p (53) 


Noise with a Gaussian Autocorrelation Function 


Noise with a Gaussian spectral intensity of bandwidth 

= 2b measured between the exp (—0.5) points of the 
power spectrum has an autocorrelation function of the 
form indicated in (385) with 


(7) = exp (—0.5b’7’). (54) 


Substituting (54) in (30), the integrals to be evaluated are 

of the form 

oe) a / cos wr exp (—dx) exp (—0.5nb’a”) dx. —- (55) 
0 


For d « b, the above integral is readily evaluated as 


I,(e) = + ae exp feo oul (56) 
Applying (56) to (80) 
Nz = 24Kb" VO. D a ie Se B..| 
1 1 on eon 
ibpetoane con ae | 
+ PO Se | ey] 
ee 
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A Note on the Estimation of Signal Waveform’ 


DAVID MIDDLETON} 


Summary—The problem of estimating signal waveform from 
received data that is corrupted by noise is briefly considered from 
the viewpoint of decision theory, in extension of some earlier work.! 
The noise is assumed to be a Gauss process, which may or may not 
be stationary. Here, however, nothing is known about the signal 
process except that it may be deterministic, entirely random, or a 
mixed process. Two new features in the present application are 
the representation of the signal process as a linear expansion (M. S.) 
in complete orthonormal sets, and suitable choices of these sets. 
Examples involving discrete and continuous sampling on a finite 
interval, with various choices of a priori distributions of signal 
parameters are described, including calculations of Bayes and 


Minimax risks. 
A the extraction of signals from noisy backgrounds 
under conditions of optimality or near optimality. 
A general theoretical framework for treating such problems 
is provided by the methods of statistical decision theory” 
as recently adapted and extended to the communication 
process.'’* The purpose of the present paper is to describe 
briefly some new results and modifications of existing 
approaches which appear useful in the operation of an 
adequate theory, vzz., 1) for optimum structure and its 
physical realization, 2) for system evaluation, and 3) for 
comparison of optimum and suboptimum systems. 

Our present problem is the estimation of signal wave- 
form S when the signal process S(¢) is combined in some 
known fashion with a noise process N(t). Estimation 
takes place on an interval (0, 7) and is based on the 
received data V(t) S(t) @® N(t). For the cases con- 
sidered here G represents simple addition, and N(¢) is a 
normal process with zero mean and covariance function 
Ky(t, , t:) = N(t)N(t.), while N(¢) need not be stationary. 
Here nothing is known Sear S(t), except that S is present 
in V and that S may be: a) deterministic, b) entirely 
random, or c) a mixed process. Two new elements of the 
present approach are: 1) the explicit representation of the 
signal as a linear expansion in a (real) complete ortho- 
normal set, ¢.g., 


CENTRAL problem in communication theory is 


S(t) =p ao, (t) (1) 


k=1 


* Manuscript received by the PGIT, January 15, 1959. 

7 Rad. Lab., The Johns Hopkins University, Baltimore, Md. 

1}, Middleton and D. Van Meter, “Detection and extraction 
of signals in noise from the point of view of statistical decision 
theory,’ J. Soc. Ind. Appl. Math., vol. 3, pp. 192-253, December, 
1955; vol. 4, ch. 4, sec. 2.2, pp. 86-119, June, 1956. For a preliminary 
account, see also D. Van Meter and D. Middleton, “Modern sta- 
tistical approaches to reception in communication theory,’’ IRE 
TRANS. ON INFORMATION THEORY, vol. IT-4, pp. 119-145; Sep- 
tember, 1954. 

2A. Wald, “Statistical Decision Functions,”’ 
Sons, Inc., New York, N. Y.; 1950. 

3 J). Middleton, ‘‘Random Processes, Signals, and Noise—An 
Introduction to Statistical Communication Theory,”’ in “‘Pure and 
Applied Physics,” Int. Ser., McGraw-Hill Book Co., Inc., New 
York, N. Y., ch. 21; in press. 


John Wiley and 


where {¢,} are orthonormal on (0, 7), @.¢., fo ¢.(Oo.() dt = 
dni 5 (Oye = 1s dy, = 0, b 4a), ands) ana ppropr te 
choice of the }¢,}, usually on the basis of the noise back- 
ground, e¢.g., the {¢,} are solutions of the homogeneous 
Fredholm integral equation (of the second kind): 


/ Ky(t, Ud, (w) du = hidx(b) , (0 < l = Dy (2) 


aii, =, 


with A, the (real) eigenvalues corresponding to the (real) 
eigenfunctions ¢, (The noise covariance Ky is also 
assumed to be continuous and positive definite, so that 
the \, are all positive.) The {a,} are thus a set of (time- 
independent) random parameters, whose statistical prop- 
erties determine those of the signal process S(t), cf. (1). 
Specifically, from (1) we have 


: S(b)¢,(2) dt. (3) 
Similarly, we have for the received-data process 
Vi) = Y eid, with ¢, = ib V(do.(b) dt, (4) 
and for the noise 


Dene ne 


b=) 


MO) i} N(t)d(t) dt, (5) 


where, (1), (8)-(5) are, of course, stochastic integrals, 
defined M.S. in the usual Riemannian sense.’ Thus, we 
have also c, = a, @ b, . The use of orthonormal expansions 
like the above is, of course, not new,’ but such devices do 
not appear to have been previously exploited in any 
general approach to signal extraction problems.” 


QuapRATIC Cost FUNCTION 


Let us now estimate S(/) at a time 4, , (0 < 4 < T), 
with an optimum system using the quadratic cost function 


(QCF) 


4M. S. Bartlett, “An Introduction to Stochastic Processes,”’ 
Cambridge Univ. Press, Cambridge, Eng., sec. 5.11; 1955. 

5 See, for example, W. H. Huggins, ‘ ‘Signal theory,” IRE Trans. 
on Crrcvrr Tuerory, vol. CT-3, pp. 210-216, December, 1956 for 
remarks on signal representations; Ge Grenander, ‘Stochastic 
processes and statistical inference,” Arkiv. Matematik, vol. 1, sec. 
3, pp. 195-227; 1950. See Almquist and Wiksells, (Stockholm, 
Sw reden), for the use of such expansions in statistical inference. For 
noise analysis, see W. B. Davenport and W. L. Root, “Random 
Signals and Noise,’ ” McGraw-Hill Book Co., Inc., New York, Nex 
chaps. 6, 14; 1958. Also, Middleton, op. cit., ch. 8 

6 An exception is an article by EK. M. Glaser ee J. H. Park, Jr., 
“On signal parameter estimation,’ IRE Trans. on INFORMATION 
Tueory, vol. CT-4, pp. 173- 174; December, 1958. These authors 
consider’ some special cases of the results summarized in their 
article, using (1) and the methods of Middleton and Van Meter, 
op. cit., and Middleton, op. cit. 


mizes the average risk (or cost) R, = Eys{C(S, , y)}. 
Following the usual minimization procedure we readily 
find for (1) in (6) that 


co 


yt = 2 VV) b.() = 


b=] 


»» aXdy(ty), (7) 


where the set of Bayes estimators {a*} of {a,}, based on 
V in (0, 7), is explicitly 
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CUS, y) = Co[S(@) — v*(Sx | V)P é : i 
(S, ) AS ae Ie, (6) U” = limv® = | o,(t) X(t) dt (12a) 
vhere y% is the optimum or Bayes estimator of S(f), er 10e 
ased on the (for the moment) discretely sampled data Bak ays. Fc 
= [V(i.), --- , Vt] in (0, 7). By definition, y* mini- oe he ®,"(V) = i VOX) dt, (12b) 


where X, is the solution of 


T 


‘{ : Ky(t,w) Xu) du = ¢.0),. (0 — =f 1) 26) 


At this point it is convenient to choose the {¢,} to 
obey (2), 7.e., to be the eigenfunctions of the noise process 
(5) on (0, 7). Then we have directly 


where a = [a@,, @, ---] is an infinite element column 
vector, like — = [£,, &, ---]. Here o is the a priori distri- 
\bution density (d.d.) of the {a,} and F,,(V | a) is the nth 
‘order conditional d.d. of V, given a, which is obtained 
(directly from (1) and F,,(V | S), the corresponding nth 
order d.d. of V, given S. 


Gauss Noise 


| Eq. (8) applies for V = S @ WN and general noise 
statistics. Let us specialize to the familiar case of additive 
normal noise. Then 


Fv S) = @n) (det K;) 
-exp {—(V — §)Ky'(V — S)/2}. 


(9) 


Writing the following (row) vectors (of nth and infinite 
orders), with 7 = 1, --- ,n, 


Se |Deae ee = 12." P=a|V Ky oe | 
k 
Dae) 


and for the infinite order square matrix W'""’, setting 


(10a) 


Ww? = WP); Var = Ky’ = WP, (0b) 
we may write (8b) generally as 
at(V) = -1 ae log a(a) 
dé, (a) (11) 


-exp {a(i—é + ®”) — 440 a} da |z-o 


for Gaussian noise backgrounds. 

With continuous sampling on (0, 7) the a% become 
functionals of V(t), and taking the appropriate limits as 
n — © of the various quadratic forms ®,”, V7 above’, 


we obtain 


7 For a justification of the formal procedure, see Middleton, op. 
cit., sec. (19.4-2). 


i| Vaglo(ay 5 «++ Ge eV ca ee damdg meee 
fat} = 2 (8a) 
[ Hs 8te sy Oy aE AV a, 2) dada 
(4) 
or more compactly, for each a, ae 
jax = —i i log / o(a)F,(V | a) dae'*® (8b) so that 
LE, (4) —=0 » 
n= bn; & =A (V(b), (14) 


where c, is given by (4). Writing Ar = [),6;:], ¢ = [ex], 
we can express (11) now for continuous sampling in the 
case of Gaussian noise as 


5 VV ee ee 
(PAID) ST log she a(a) (15) 


-exp {a[c— + Az'c] — 34A7 a} da |p . 


STRUCTURE 


Before listing results for special choices of o(a), let us 
remark on the structure of the optimum systems implied 
by (15). Letting ¢, be the weighting function h of a linear, 
relizable filter, e.g., 


we observe that 
: 
nes | Viih,(T — 2) dt, (17) 
0 


so that c, is the output of said filter at ¢ = T’.. The opera- 
tions on c, implied by (15) can then be obtained by 
computation, depending on o(a), once the j{c,} have 
been found. The Bayes waveform estimator with QCF 
here consists therefore of a parallel bank of linear, realiz- 
able filters, cf. (16), each output of which goes to a suitable 
computer, whose output in turn is the desired a¥,. These 
a*, may then be multiplied by ¢,(4,) and added to give 
the Bayes estimate of the waveform at 4, e.g., 


S*(h) = Ds Ph) 


By allowing 4, to vary in (0, 7), the entire waveform in 
that interval may be reproduced (in estimate). The ¢;, 
of course, depend on the (known) spectral distribution 
of the background noise through (2). 
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SPECIAL RESULTS 


Here we shall give the Bayes estimators for some useful 
choices of (a): 


Case I: Gauss d.d. of {a,}. 


o(a) = K, exp {—1(4 — a)A ‘(a — a)}, (18) 
where A = A and A in positive definite. 
Eq. (11) yields 
ax = [we + A) "(@™ 4+ AA)), (19) 


which for continuous sampling and the ¢, of (2) becomes 
Gin = (Ap -b A.) (Age Asa) ee (20) 


If A = [o;6,,], 7.e., the a, are independent, (20) reduces 
at once to 


Fe Cee Gids/ 1 
kT 1 = i/o ? 


which is a generalization of Glaser and Park’s result, (16).° 


[=a a ea) 


Case II. Independent Rayleigh d.d. of {a,}. 
Here each a; obeys 


a, exp {—a;/20;} 
2 


o,( Ay) = : ) a; = 0. (22) 
The Bayes estimators are (for continuous sampling) 
BG es Wri, 
pee enw = 5) 
(ARH 4 1 =) eos + 2x)e"O(@)] 
aVre* (1 + O(a] +1 
with 
x = (,/V2d.)/(L + dz/oi)! (23a) 


where ,/, is the usual confluent hypergeometric function 
and @ is the error integral 


Qs) = ==) < arte 


Case III: Independent unsymmetrical, uniform d.d. of {a,}. 
For this case 
ox (Ax) = P;?, 


O08 Pe (24) 


so that as P, — ©, we have again (for continuous sampling) 
atr = ¢ + (2d./me [1 + O@)]", 2 =e./V 2% . (25) 


Note that if on the average x is large, a#, ~ c, , which is 

linear in V. 

‘Case IV: Same as III; symmetrical d.d. of {a,}. 
Now o;(a;) is modified to 

—Pi/2 <a, < Pi/2, 


ox( Ax) = ee (26) 


June 
with the result that (15) becomes simply” 
(27) 


Note that only in Cases I and IV, for independent a,, is 
a*, a linear functional of the data V. 


ib = Ox « 


SimpLeE Cost FuNcCTION 


a3 


. . 9 
strict’? version 


-T] 5S; =; v)} 


Bayes estimators here are 


We consider here the 
C(S, y) = C4 An (28) 


for continuous estimators. 


(unconditional) maximum likelihood estimators.’ Writing 

the likelihood functions 

L(V | a) = o(a)F,(V | a); 1(V | a) = F,(V | a) (29) 

the solutions of 

(UMLE):5* =0;) (CMLE): a =0 (80) 
Os ap=ak* Gk |an=dr 


yield respectively unconditional maximum likelihood 
estimators (UMLE) and conditional maximum likelihood 
(CML) estimators of the a,. The Bayes estimator 7* of 
waveform S at f, is again given here by (7); however, the 
a* are now replaced by 4%. The conditional estimator 4 5 
at t, is likewise 


= ME, di (t). (30a) 


For additive Gauss noise we readily obtain from (9)— 
(10b) and (30): 


i =1 Oo (n) n 
(MLE): «2 Sey = > as “| op 


=.0; alle; 


as the condition of a UMLE of a,. With continuous 
sampling and the appropriate {¢,}, cf. (2), (31) becomes 


2 0G. =i 
cumrE):| «* 2 ees Pos (c, = Ay) Nx | ’ = (I) (32) 
The conditional MLE for (82) is at once 
hr = c(V), (33) 


e.g., just the outputs of the bank of linear filters (16), (17). 
For Cases I-IV considered above, we have specifically 


“f Cy +. Gyd;,/ 0% 
1 + di / ox 


a(1 ah dz /on) (1 = 


(34a) 


= Air lr, QcF 


ce + 4d, + 4dz/0%) (34b) 


Gir le =a 


(34c) 


Gir lan = at liv =c, = d, Harry . 


8 Glaser and Park’s result, (10). 
® Middleton and Van Meter, op. ctt., cf. (4.2). 


te again that it is possible for different cost functions 
yield the same Bayes estimators [cf. (34a), (34c), and 
iddleton and Van Meter’]. (The associated Bayes risks, 
ever, are not necessarily the same.) 


System EVALUATION AND COMPARISON 


This is accomplished in the usual fashion by computing 
e (Bayes) risk for optimum systems and the average 
sk for suboptimum ones and then comparing the results 
s-d-vis average costs, or say, input signal-to-noise ratios 
or the same average cost), etc. Here it is often convenient 
extend the measure of average risk associated with 
e estimate of S at 4, to include all 4 in (0, 7). We 
‘cordingly define a time-averaged risk (TAR) by 


” 
Reap [ BMH) dh (35) 
0 

r Bayes systems), with R* and R*(é,) replaced by Rr, 
(4) for suboptimum cases. Note that since 0 < R*(t,) < 
H(2,), all 4, then RF < Rp. 

‘We illustrate these remarks with several results based 
n the QCF and continuous sampling. Applying (6) and 
'), we get 


T —, 
Rt Z [S* — 28y¥ + y2"] dt, (86a) 
0 
Co 2 * 2 
= T ») [a, — 2a,a%r + ax]. (36b) 


Ss 


ecify o(a). Here we choose Cases IV and I and find 
a straightforward way that 


6, 

RE = qr LL = Cow, (37) 
) C d 
=-2y(_,) 
| Bile =p 2p aye oe) 
ee 
P 
ys = ih Ky(t, t) dt (38a) 


=; w= N*, (N= 0). 


‘or Cases II and III, R* is much more involved. Observe 
hat for white noise backgrounds, R% |;y -> ©, since \, = 
V,/3, all k, where W, = spectral intensity density of the 
oise: while R*¥|, may be finite if o; is such that the series 
38) converges. 


0 Middleton: A Note on the Estimation of Signal Waveform 89 


REMARKS 


This approach, using a linear development of the form 
(1) for the signal, is a useful alternative to the direct 
evaluation of waveform, based on the a priori statistics 
of S, e.g., c(S). The results are equivalent, because of the 
linear character of (1). However, it may be simpler to 
instrument the system when (1) is used [ef. (3)]. The 
same class of a priori knowledge as to the signal statistics 
is still required; here we need a(a), [cf. (8), e¢ seq.]. In 
practice, most of the time the receivers do not know o(a) 
for o(S)], so that a subsidiary extremal principle must be 
invoked if we are to apply the decision theory approach. 
This is reasonable, however, and two such principles are 
minimax and maximum average wncertainty."° Both result 
in essentially “worst”? best (7.e., Bayes) systems with 
respect to “most unfavorable” a priori distributions of 
a (or S). The uniform d.d. of Cases III and IV are minimax 
(with QCF), as well as yielding maximum average uncer- 
tainties with respect to maximum values.” 

The development (1), however, does not appear to 
offer any advantages when S has known waveform but 
one or more unknown parameters, eg., S = S(t; 6) 
which are to be estimated. Then a, = a;,(8), and in general 
we cannot invoke independence of the a,. Furthermore, 
for the simple cost functions it is not possible to obtain 
explicit forms of the estimators when the 6 appear trans- 
cendentally in S, 2.e., as a frequency, a phase, a delay, etc. 

Although we have discussed interpolation (7.e., filtering) 
only, the procedures of the above are easily extended 
to extrapolation (e.g., prediction, etc.) if we represent 
the extrapolated waveform as S(f,) = oS: didi(tr), ty 
outside (0, 7’), where the d, are random variables, now 
(in general) statistically dependent on the a, of S() = 
doe Udi(t), (0 < t < T), developed on the interval 
(0, 7). The details are left to a subsequent study. 

Finally, we note as an example of a case where the o(a) 
(or o(S)) are known a priori that (1) may be used in 
communication systems where the S’s are constructed at 
the transmitter by one or a set of random mechanisms 
which select the a,, (k = 1, --- m, say), with known 
statistics and with given {¢,}. These {¢,} are also known 
at the receiver, as are the statistics, o(a), of the a,. The 
receiver then adopts an optimization principle and makes 
“‘best”’ estimates of the corrupted waveforms V = S + N 
actually received, using the above information. The 
accuracy of such a system is being studied, as are other 
cost functions, 7.e., Bayes and Minimax risks for the 
maximum likelihood estimators, and the explicit com- 
parisons of various optimum and suboptimum systems. 


10 This maximizes — log o(a) subject to maximum value con- 
straints on the a, or fixed mean-square values, P,. The former 
yields the independent, uniform distribution of Cases III and IV, 
while the latter yields the Rayleigh d. d. of Case II. 

11 Middleton, op. cit., ch. 21. 
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Cumulative Distribution Functions 
for a Sinusoid Plus Gaussian 
Noise* 


In certain problems which arise in ap- 
plications of communication theory, it is 
desirable to know the probability that a 
variable lies within a given amplitude range. 
A random variable which occurs frequently 
in such cases is one consisting of a sinusoidal 
signal (with random phase) corrupted by 
uncorrelated additive Gaussian noise. Let 
x(t) be such a variable: 


a(t) = s(t) + n(0) (1) 
s(t) = Asin (wt + ¢) (2) 
n(t) = Gaussian random variable (3) 
Ein(t)} = 0 (4) 
Ege (fy awe. (5) 


The phase angle, ¢, is uniformly distributed 


between 0 and 27 radians. 


In order to determine the probability 


is necessary to know the cumulative prob- 
ability distribution for 2x(¢) since 


b 
Pax ee / ieee 
=P (Oi En O)ae mB) 
where p(x) is the probability density 


function for « and P,(x) is the cumulative 
probability of x 


Pa) = | p.le) de 


S. O. Rice! has shown that the prob- 
ability density function for x(¢) [as defined 


by (1) through (5) above] is given by 


1 


a WV 20 Wo 


pAx) = 


fo exp {—(x — A cos 6)*/2y} dé. 


(8) 
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It does not appear that this result can be 
presented in terms of tabulated functions 

Before operating on (8) to obtain P,(2x) 
it is convenient to transform x to a variabl 
of unity variance 


x \ 
Opes (9) 
Oy 
JN 
ao -Elr}=vwt+s Ce 


so that (6) becomes 


Pa<e<v =P <y< 


Oo; 


-r(t)~r(2) 


It is also helpful to define a signal to noise 
ratio, 7 in the usual sense 


(11) 


that 2 lies between two values, a and 8), it 2 2 
; : 1S. O. Rice, ““Mathematical Analysis of Random E\s } A 
Noise,’’ Bell Telephone Sys. Monograph B-1589, eq. ff => Soy = : (12) 
* Received by the PGIT, November 24, 1958. 3.10-6, p. 105; 1945. E{n*} = Wo ’ 
CuMULATIVE DIsTRIBUTION FUNCTIONS 
FOR THE SUM OF A SINUSOID AND A GAUSSIAN VARIABLE 
Values of Py(y) — % 

y mnt) 7 2 20 area pis r =2 r=3 r=4 r=6 =8 r = 10 r = 12 (ein I} r = 20 f = co 
0.00 0.00000 | 0.00000 | 0.00000 | 0.00000 | 0.00000 | 0.00000 | 0.00000 | 0.00000 | 0.00000 | 0.00000 | 0.00000 | 0.00000 | 0.00000 0.00000 
0.10 0.03983 | 0.03958 | 0.03860 | 0.03636 | 0.03220 | 0.02936 | 0.02757 | 0.02569 | 0.02481 | 0.02431 | 0.02400 | 0.02369 | 0.02339 0.02253 

O20 0.07926 | 0.07877.| 0.07687 | 0.07255 | 0.06450 | 0.05896 | 0.05542 | 0.05164 | 0.04983 | 0.04881 | 0.04816 | 0.04754 | 0.04693 0.04517 
| 0.30 0.11791 | 0.11721 | 0.11449 | 0.10837 | 0.09696 | 0.08901 | 0.08383 | 0.07811 | 0.07529 | 0.07369 | 0.07267 | 0.07169 | 0.07075 0.06804 
0.40 0.15542 | 0.15455 | 0.15117 | 0.14363 | 0.12960 | 0.11968 | 0.11303 | 0.10538 | 0.10145 | 0.09918 | 0.09774 | 0.09635 | 0.09502 0.09128 
0.50 0.19146 | 0.19046 | 0.18661 | 0.17811 | 0.16236 | 0.15104 | 0.14321 | 0.13377 | 0.12864 | 0.12560 | 0.12363 | 0.12175 | 0.11997 0.11503 
0.60 0.22575 | 0.22466 | 0.22035 | 0.21159 | 0.19510 | 0.18304 | 0.17444 | 0.16352 | 0.15722 | 0.15331 | 0.15072 | 0.14822 | 0.14584 0.13947 
0.70 0.25804 | 0.25692 | 0.25276 | 0.24383 | 0.22757 | 0.21552 | 0.20665 | 0.19482 | 0.18752 | 0.18275 | 0.17947 | 0.17620 | 0.17304 0.16482 
0.80 0.28814 | 0.28703 | 0.28302 | 0.27459 | 0.25947 | 0.24817 | 0.23963 | 0.22767 | 0.21980 | 0.21434 | 0.21040 | 0.20630 | 0.20216 0.19139 
0.90 0.31594 | 0.31490 | 0.31119 | 0.30363 | 0.29042 | 0.28054 | 0.27292 | 0.26181 | 0.25404 | 0.24833 | 0.24398 | 0.23919 | 0.23400 0.21958 
1.00 0.34134 | 0.34040 | 0.33714 | 0.33074 | 0.32000 | 0.31208 | 0.30591 | 0.29664 | 0.28982 | 0.28453 | 0.28029 | 0.27533 | 0.26949 0.25000 
1.10 0.36433 | 0.36352 | 0.36080 | 0.35575 | 0.34781 | 0.34219 | 0.33786 | 0.33125 | 0.32622 | 0.32214 | 0.31872 | 0.31448 | 0.30905 0.28367 
1.20 0.38493 | 0.38426 | 0.38214 | 0.37851 | 0.37346 | 0.37026 | 0.36793 | 0.36448 | 0.36184 | 0.35962 | 0.35769 | 0.35517 | 0.35166 0.32251 
1.30 0.40320 | 0.40268 | 0.40118 | 0.39895 | 0.39665 | 0.39576 | 0.39538 | 0.39512 | 0.39502 | 0.39495 | 0.39486 | 0.39467 | 0.39426 0.37120 
1.40 0.41924 | 0.41888 | 0.41797 | 0.41703 | 0.41716 | 0.41827 | 0.41958 | 0.42207 | 0.42422 | 0.42606 | 0.42764 | 0.42966 | 0.43233 0.45483 
1.50 0.43319 | 0.43298 | 0.43261 | 0.43278 | 0.43491 | 0.43757 | 0.44013 | 0.44460 | 0.44829 | 0.45139 | 0.45405 | 0.45741 | 0.46188 0.50000 
1.60 0.44520 | 0.44512 | 0.44522 | 0.44630 | 0.44990 | 0.45360 | 0.45693 | 0.46245 | 0.46681 | 0.47035 | 0.47229 | 0.47689 | 0.48142 
1.70 0.45543 | 0.45547 | 0.45595 | 0.45771 | 0.46225 | 0.46648 | 0.47010 | 0.47580 | 0.48004 | 0.48331 | 0.48589 | 0.48886 | 0.49226 
1.80 0.46407 | 0.46420 | 0.46497 | 0.46719 | 0.47219 | 0.47670 | 0.48000 | 0.48520 | 0.48880 | 0.49137 | 0.49326 | 0.49527 | 0.49728 
1.90 0.47128 | 0.47149 | 0.47246 | 0.47492 | 0.47997 | 0.48402 | 0.48713 | 0.49144 | 0.49414 | 0.49591 | 0.49711 | 0.49824 | 0.49920 
2.00 0.47725 | 0.47751 | 0.47860 | 0.48113 | 0.48591 | 0.48947 | 0.49204 | 0.49532 | 0.49716 | 0.49824 | 0.49889 | 0.49943 | 0.49980 
2.10 0.48214 | 0.48242 | 0.48356 | 0.48602 | 0.49032 | 0.49328 | 0.49528 | 0.49759 | 0.49872 | 0.49931 | 0.49962 | 0.49984 | 0.49996 
2.20 0.48610 | 0.48640 | 0.48752 | 0.48981 | 0.49351 | 0.49585 | 0.49731 | 0.49883 | 0.49947 | 0.49975 | 0.49988 | 0.49996 | 0.49999 
2.30 0.48928 | 0.48958 | 0.49065 | 0.49269 | 0.49576 | 0.49753 | 0.49853 | 0.49946 | 0.49980 | 0.49992 | 0.49997 | 0.49999 | 0.50000 
2.40 0.49180 | 0.49209 | 0.49307 | 0.49485 | 0.49730 | 0.49857 | 0.49923 | 0.49977 | 0.49993 | 0.49998 | 0.49999 | 0.50000 
2.50 0.49379 | 0.49406 | 0.49493 | 0.49642 | 0.49832 | 0.49921 | 0.49962 | 0.49991 | 0.49998 | 0.49999 | 0.50000 
2.60 0.49534 | 0.49558 | 0.49634 | 0.49756 | 0.49898 | 0.49957 | 0.49982 | 0.49996 | 0.49999 | 0.50000 
} 2.70 0.49653 | 0.49675 | 0.49739 | 0.49837 | 0.49940 | 0.49978 | 0.49992 | 0.49999 | 0.50000 
2.80 0.49744 | 0.49763 | 0.49816 | 0.49892 | 0.49966 | 0.49989 | 0.49996 | 0.50000 
2.90 0.49813 | 0.49829 | 0.49872 | 0.49930 | 0.49981 | 0.49995 | 0.49998 
3.00 0.49865 | 0.49878 | 0.49912 | 0.49956 | 0.49990 | 0.49998 | 0.49999 
3.10 0.49903 | 0.49914 | 0.49941 | 0.49972 | 0.49994 | 0.49999 | 0.50000 
3.20 | 0.49931 | 0.49940 | 0.49960 | 0.49983 | 0.49997 | 0.50000 
3.30 0.49952 | 0.49958 | 0.49974 | 0.49990 | 0.49999 
3.40 0.49966 ; 0.49972 | 0.49983 | 0.49994 | 0.49999 
3.50 0.49977 | 0.49981 | 0.49989 | 0.49996 | 0.50000 
3.60 0.49984 | 0.49987 | 0.49993 | 0.49998 
lars LO 0.49989 | 0.49991 | 0.49996 | 0.49999 
| 3.80 0.49993 7. 0.49994.) 0.49997 | 0.49999 
| 3.90 0.49995 | 0.49996 | 0.49998 | 0.50000 
4.00 0.49997 | 0.49998 | 0.49999 


Py(y) = 4% + JF puy) ay 
Pu(y) = probability density function for y 
r = signal to noise ratio 


veriance of sinusoid 


variance of Gaussian variable 


Eyy?; = ] 
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then follows by means of straightforward 
ansformations that 


Wy) =5 +2 f tet a tny 
— (2r)'” cos 6} 
me ert {—(2r)™ c0s.6})'de. (18) 


his integral has been evaluated on an 
3M 704 computer using parabolic integra- 
on with an integration interval of 2.5°. The 
sults are tabulated (see preceding page) 
3 P,(y) — 1/2 vs y for various values of r. 
he accuracy of this table was checked by 
subling the integration interval and re- 
»mputing the table. The results agree to 
, least the five digits tabulated. For com- 
eteness of the table, the cases r = 0 and 
= o have been included although r = 0 
elds simply the error function and 7 = 
rresponds to the cumulative arcsine dis- 
ibution. 
A. LEVINE 
R. B. McGuerr 
Systems Dev. Labs. 
Hughes Aircraft Co. 
Culver City, Calif. 


a the Characteristics of Error- 
orrecting Codes* 


L 
t 
f 


In his paper,! Sacks pointed out that in 
‘der for a code to correct e-errors, no set 
2e or less characteristics should be 
early dependent (mod 2), and _ con- 
srsely. This fact and its implications may 
> illustrated more clearly with the aid of 
yme basic notions of group theory and 
lapping. 

| When n characteristics are written as 
ylumn vectors of a matrix r by n, they 


20,034) 2-1 On| 


(1) 


= [A, A203 Sein Ay | u,| 


ecoding is accomplished by multiplying 
ie received message J by A. Thus, 


Wisaee (2) 


here & is a column matrix of r components. 
rror correction is achieved by associations 
tween the 2” possible versions of R and 
e various linear combinations of the 
lumn vectors of A. 

As the checking bits X7 of J are deter- 
ined by satisfying the equation 


ol 
allie 


* Received by the PGIT, April 14, 1959. 

1G. Sacks, ‘Mul tiple error correcting by 
ans of parity checks, ” TRE Trans, on INFORMA- 
nN THEORY, vol. Tes pp. 145-147; December, 


(mod 2) (3) 
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where Y;, represents the information bits, 
the code points are mapped into the null 
vector. Hach of the 2” possible points are 
mapped to some linear combination of the 
n-characteristics. This is, in fact, a many 
to one mapping, and the number of points 
associated to each linear combination is a 
multiple of the size of the code. 

If a set of s characteristics are 
on each other as 


Sere er0 


t=1 


dependent 


(mod 2), (4) 


then we have 


(mod 2) b+ec=s. (5) 


Thus the errors corresponding to a = 5>°_, ai 
are indistinguishable from the errors cor- 
responding to a = yes a; . Therefore, only 
one of these two sets of errors can be cor- 
rected and the other set will always slip by 
undetected. The alternative is to say that 
in this case errors can only be detected 
and not corrected. 

Thus the problem of synthesizing a code 
is the same as finding a set of vectors such 
that this set of vectors and some of their 
linear combinations covers the 2” possible 
nonzero vectors in a prescribed manner. 
A lossless code is one for which the set of 
vectors, and their linear combinations up 
to a certain number of vectors, covers the 
2” vectors in a one to one manner. 

One may also visualize this situation as 
that of partitioning the 2” points into 2* 
geometrically similar partitions where each 
partition contains one code point. 

The shape of the partition is determined 
by the n-characteristics. The lossless code 
is achieved when these partitions are per- 
fect spheres. The single error correction 
lossless code is particularly easy to con- 
struct as one simply writes down the 2” — 1 
nonzero vectors as the characteristics. In 
the general case of the Galois Fields 
(p™” — 1)/(p™ — 1) nonzero vectors are 
used as shown by Cocke’. 

The dependent relationships may be 
used to calculate the @ values of Slepian 
directly without resulting to cosets. The 
method of calculation is illustrated by an 
example of the (8, 3) code with the char- 
acteristic matrix. 


Cee OO) Oe Os, Oye Oe 
el Pelee Or Ore el) 
ie law) ae Oe eel Or 0) 
(ee cela Ol Osun tee. 0) 
iO eer 0 nO Oe te sO 
OR Oe Ome Oc) at 
2J, Cocke, “Lossless symbol coding with non- 


primes,’ IRE Trans. ON INFORMATION THEORY, 
vol. IT-5, pp. 33-34; March, 1959. 
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It is seen that the following dependent 
relationships are present: 


a, + a; +4, + a3 = 0 
a; + aj + af + af = 0 (mod 2). (6) 
d, + a; + a2 + a; = 0 


All other such relationships will involve at 
least five vectors. In order to increase the 
Slepian Q value, correcting a double error 
is always preferable to the correction of a 
triple error. Therefore, these dependent 
relationships can be neglected. A tabulation 
of double errors immediately gives the o’s 
of the code, as shown in Table I. 


TABLE I 


Error Corrected 


Errors Not Corrected 


D1 5/ 
Pit 1/5/ 
yf 12/ 

31’ 3/5’ 
5/ 1/3’ 
PA 2/3" 

/ 23’ 

33/ 


On the other hand, there is a close re- 
lationship between the degree of dependency 
of the characteristics and the minimunt 
distance of the code. 

Proposition: The minimum distance of 
a group code is d if and only if every set of 
d — 1 vectors of the characteristic matrix 
is independent. 

Proof: We shall prove that having a set 
of d linearly dependent vectors implies the 
existence of a pair of code points at a 
distance d apart and conversely. 

First of all, one must show the existence 
of a characteristic matrix for each group 
code which can be shown in the following 
manner, Given any group code where each 
code is written as a column vector, it is 
possible to obtain a sub-matrix from this 
matrix that consists of k columns while 
the rows corresponds to information posi- 
tion form a unit matrix. Then the k checking 
position vectors are the desired character- 
istics. This proves the existence of a char- 
acteristic matrix for every group code. 

Suppose there exists a set of d vectors 
of the n-characteristic matrix which is de- 
pendent. This means that given any code 
point if d errors corresponding to these d 
vectors are present, the detector will print 
zero. Therefore, it is clear that this message 
with d errors coincides with another point. 
So, there is at least one pair of coding 
points at a distance d away. 

It is also clear that if the distance be- 
tween some pair of code points is d, then 
there is at least one set of d vectors which 
are linearly dependent. 

A lossless code which is e-error correcting 
is obtained if one can find a matrix where 
each set of 2e vectors is linearly independent, 
and if for each set of e vectors there exists 
at least one set of e + 1 vectors such that 
this set of 2e + 1 vectors are dependent. 


Rosert CHIEN 
IBM Research 
Yorktown Heights, N. Y. 
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Phillip Bello (S ’52—A ’55) was born in 
Lynn, Mass. on October 22, 1929. He re- 
ceived the B.S. in electrical engineering 
from Northeastern 
University, Boston, 
Mass., in 1953 and 
the S.M. degree in 
electrical engineering 
from the Massachu- 
setts Institute of 
Technology, Cam- 
bridge, Mass., 
in 1955. His Master’s 
thesis was concerned 


= with characterizing 

P. BeLto aa sonal 
the return signa 

in a phase com- 


parison radar caused by a complex scin- 
tillating target. From 1955 to 1957, he 
was an assistant professor of Communica- 
tions at Northeastern University, teaching 
“Transients in Linear Systems” and ‘“TFil- 
tering and Prediction’ in the graduate 
school. He reentered M.I.T. in September, 
1958 to start the Sc.D. program in electrical 
engineering and is currently working on his 
thesis which is concerned with the applica- 
tion of linear transformation theory to the 
synthesis of linear, active and nonbilateral 
networks. 

While at Northeastern University, Mr. 
Bello was involved in research on the IFF 
problem, which necessitated the use of 
statistical communication theory. From 
1956 to 1958, he was employed by Dunn 
Engineering Associates, Cambridge, Mass., 
where he was engaged in analytical studies 
associated with various types of radar 
systems frequently involving statistical 
problems such as detection, and measure- 
ment of radar parameters of targets. 
During the summer of 1958, he was em- 
ployed at the Applied Research Laboratory 
of Sylvania Electronic Systems, Waltham, 
Mass., and worked on problems in statisti- 
eal radar theory. 

Mr. Bello is a member of Tau Beta Pi, 
Sigma Xi, and Eta Kappa Nu. 
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Herman Blasbalg (A ’48—M ’55— 
SM ’56), was born on June 17, 1925 in 
Poland. He received the B.E.E. degree from 
the College of the 
City of New York, 
Wha, Nog shiny aiMoyzbe 
the M.S. degree 
in electrical engi- 
neering from the 
University of Mary- 
land, College Park, 
in 19152 >and the 
Doctor of  Engi- 
neering degree from 
The Johns Hopkins 
University, _ Balti- 
more, Md., in 1956. 

From 1948 to 1951, he was employed by 
Melpar, Inc., Alexandria, Va. During this 
time he was involved in research design and 


H. BuasBaLGe 


development of a ppm communications sys- 
tem and voice channel compression as well 
as other applied information theory pro- 
jects. From 1951 to 1956, he was employed 
by the Radiation Laboratory of The Johns 
Hopkins University where he was project 
supervisor of the group in charge of de- 
veloping automatic airborne signal analyzer 
systems. Dr. Blasbalg also worked on 
statistical detection theory and on a mathe- 
matical theory of observation. In 1956 he 
became a research scientist and staff con- 
sultant. In October, 1956, he joined the 
staff of Electronic Communications, Inc., 
Baltimore, Md., and is presently involved 
in applied statistical decision theory, ob- 
servation theory and automatic observing 
systems. 

Dr. Blasbalg is a member of Sigma Xi 
and the Institute of Mathematical 
Statistics. 


Marvin Blum (M 756) was born on June 
18, 1928 in New York, N. Y. He received 
the B.S. 


degree from Brooklyn College, 
N. Y., in June, 1948, 
and has taken gradu- 
ate courses in mathe- 
matics, physics, and 
electrical engineering 
from George Wash- 
ington University, 
American University, 
Maryland Univer- 
sity, the National 
Bureau of Standards 
School, and the Uni- 
versity of California 
at Los Angeles Ex- 


M. Buum 
tension. 

He worked at the National Bureau of 
Standards in the Central Radio Propaga- 
tion Laboratory until 1950. He then trans- 
ferred to the Ordnance Division, where he 
conducted radar reflection studies relating 
to missile proximity fuzes. Since July, 1954, 
he has been employed at Convair, San 
Diego, Calif., where he is conducting 
theroretical investigations in smoothing 
and prediction filters, noise simulation, and 
data reduction. Presently, he is working in 
the newly organized Convair Astronautics 
Division. 

Mr. Blum is a member of the Society for 
Applied Mathematics. 


Charles V. Freiman (M 757) was born in 
New York, N. Y., on June 17, 1932. He 
received the A.B. degree in 1954 from 
Columbia College, New York, N. Y., and 
the B.S. degree in 1955 and the M.S. degree 
in 1956 from the school of engineering, 
Columbia University, New York, N.Y. 

Since 1956, he has been employed as 
an instructor in electrical engineering at 


June 


Columbia Univer- 
sity. He has also 
worked as an engi- 
neer or consultant 
(in logical design) 
for Proeter and 
Gamble, Bell Tele- 
phone Laboratories, 
Electronics Research 


le | Laboratories (Co- 
r lumbia University), 
VB RBLMAN International Busi- 


ness Machines Cor- 
poration, and the Allard Instrument Corpo- 
ration. 

At present, he is working toward the 
Se.D. degree in electrical engineering at 
Columbia University. 

Mr. Freiman is a member of Tau Beta Pi 
and Eta Kappa Nu. 
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Janis Galejs (A ’52), for a photograph 
and biography, please see page 131 of the 
September, 1958 issue of these TRANS- 
ACTIONS. 

Arthur Gill was born in Haifa, Israel, on 
April 18, 1930. He attended the Massachu- 
setts Institute of Technology, Cambridge, 
Mass., where he re- 
ceived the B.S. and 
M.S. degrees in elec- 
trical engineering in 
1955 and 1956, re- 
spectively. He  re- 
ceived the Ph.D. 
degree in the same 
field from the Uni- 
versity of California, 
Berkeley, in 1959. 

From 1954 to 1956 
he served as a teach- 
ing assistant in the 
department of electrical engineering at 
M.1.T. From 1956 to 1957 he worked in 
the research division of the Raytheon 
Manufacturing Company, Waltham, Mass., 
where he was engaged primarily with semi- 
conductor circuitry design. Since 1957 he 
has been a member of the staff of the 
University of California, where he has 
served as a teaching associate in electrical 
engineering and later as an assistant pro- 
fessor. He is also associated with the 
Electronic Research Laboratory of the 
University, where he is working on infor- 
mation theory problems, and with the 
advanced programming development group 
of the Bendix Computer Division of the 
Bendix Aviation Corporation. 

Dr. Gill is a member of Eta Kappa Nu, 
Tau Beta Pi, and Sigma Xi. 
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William A. Janos (M ’59) was born on 
November 9, 1926, in Easton, Pa. He served 
in the U. 8. Army from 1945-1947. In 1951 


he received the B.S 
degree in physics 
from Rutgers Uni- 
versity, New Bruns- 
wick, N. J. Since 
1951 he has been 
with Convair and 
Convair-Astro- 
nautics, San Deigo, 
Calif., taking leaves 
of absence to obtain 
the M.A. degree in 
1954 and the Ph.D 
egree in 1958, both in physics, from the 
niversity of California, Berkeley. He was 
ecipient of the University’s appointed 
saching assistantship in the physics de- 
rtment, and also the Convair Scholarship 
ward. 

| He has been engaged in applied analysis 
nd spectral theory related to analytical 
namics, wave propagation and diffrac- 
on, variational techniques in least-time 
ajectories for thrust-propelled flight, con- 
rol system analysis and synthesis, noise 
heory and optimal linear estimation. He 
} presently conducting analytical research 
digital-analog communication, computa- 


W. A. JANOS 


on and control systems, and initiating 
udies in information theory of continuous 
purces. 

Dr. Janos is a member of the American 
*hysical Society. 


Julian Keilson (SM ’56) was born in 
brooklyn, N. Y., on November 19, 1924. In 
947, he received the B.S. degree in physic 
from Brooklyn Col- 
lege, N. Y., and the 
A.M. and Ph.D. de- 
grees in physics from 
Harvard University, 
Cambridge, Mass., 
in 1948 and 1950, re- 
spectively. He was 
an Atomic Energy 
Commission predoc- 
toral fellow from 
1949 to 1950, and a 
research fellow in 
electronics at Har- 
ard University from 1950 to 1952. 

'From 1952 to 1956, Dr. Keilson was 
mployed at the Massachusetts Institute 
£ Technology’s Lincoln Laboratory, Lex- 
agton, Mass. In 1956, he joined the Applied 
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